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Executive Summary 

Institutions of higher education (IHE) play an important role in math and science 
education by providing undergraduate instruction, teacher training programs, and in-service 
training for K-12 teachers. The National Science Foundation (NSF)-funded System-wide 
Change for All Fearners and Educators (SCAFE) project sought to effect change in its partner 
IHEs by creating a transformative culture through cross-cultural working teams that operated at 
the intersections among K-12 districts, colleges of education, and colleges of mathematics, 
science, and engineering (SCAFE, 2005). The SCAFE goals for IHEs are as follows: 

1. To improve science, technology, engineering, and mathematics (STEM) undergraduate 
education; 

2. To improve collaboration between STEM and education faculty regarding preservice 
programs; 

3. To improve collaboration between IHE faculty and K-12 districts regarding in-service 
training; and 

4. To improve institutional policies and practices that support these activities. 

As part of the SCALE IHE case studies line of work, this paper provides findings on the 
effects of the SCALE project at the California State University, Northridge (CSUN) between 
March 2005 and August 2007. Case studies of SCALE’S partner IHEs — ^the California State 
University, Dominguez Hills (CSUDH), and the University of Wisconsin-Madison (UW- 
Madison) — have been produced (Hora & Millar, 2007; Hora & Millar, 2008). A cross-case 
analysis of the three IHE case studies will present a diagnostic approach to evaluating STEM 
education interventions in complex organizations. 

Methodology 

This qualitative case study addresses two interrelated challenges with studying and 
evaluating change processes in complex institutional environments. First, because reform efforts 
such as SCALE coexist with and seek to change diverse levels and elements of an institution, it 
is critical to understand the context in which a reform effort unfolds (Patton, 2006; Katzenmeyer 
& Lawrenz, 2006; Anderson & Helms, 2001). Second, research on reform implementation in 
both higher education and K-12 finds that when faced with policy directives, individuals 
frequently adapt them in the process of making sense of and applying them in local environments 
(Birnbaum, 1988; Spillane, Reiser, & Reimer, 2002). These research challenges are interrelated 
because local contextual factors shape the sense-making process, as individuals are inclined — if 
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not required — ^to attend to eontextual features of their organizations, especially the cultural 
norms of administrative units such as academic departments (Gamoran et ah, 2003; Cobum, 
2001). However, research in this area is complicated by a paucity of theories and methods that 
adequately address the multilevel and recursive nature of these relationships among context, 
culture, and individual cognition. This lack is also very apparent in program evaluations. In 
response to these concerns, this study of SCALE at CSUN focuses on contextual factors by 
identifying which are salient to STEM education reform, their relationship to group- and 
individual-level cultural phenomenon (i.e., mental models), and their collective influence on the 
SCALE intervention. 

This study used a repeated cross-sectional design, with in-depth interview data collected 
at Time 1 (Tl; June 2006) and Time 2 (T2; June 2007). Other data collected for this research 
included official university and SCALE documents, and observations of SCALE meetings. Non- 
random sampling procedures were used to identify interview respondents, which included 18 at 
Tl and 16 at T2, for a total of 34 interviews with 25 unique individuals. Interviews were coded 
using a stmctured approach to grounded theory (Strauss & Corbin, 1990) based on a coding 
scheme we call the institutional context framework (ICE) (Hora & Millar, 2006). The ICE is 
comprised of the following code families: external environment, internal stmcture, resources 
(fiscal and social), pervasive values, individual sense-making, and practices. Interview data were 
coded, reduced, and inductively analyzed for recurring patterns and themes in the data in order to 
identify contextual factors salient to SCALE, group- and individual- level cultural norms, and 
individual sense-making processes. Additional analytic procedures included (a) a causal network 
analysis that allows for the graphic representation of the relationships among these multi-level 
phenomenon and (b) an exploratory analysis of mental models and their constituent cultural 
schema as situated within the institutional context of CSUN. 

A Snapshot of the Intervention Site: California State University, Northridge 

CSUN is located in the San Eernando Valley of Los Angeles County. Its location in a 
large, ethnically diverse urban area exerts a major influence on the institution’s identity, mission, 
and academic programs. CSUN is part of the 23-campus California State University (CSU) 
system and shares with its sister CSU campuses a focus on undergraduate education and 
providing affordable opportunities for a broad cross-section of California’s population. The fall 
2007 enrollment at CSUN was approximately 35,000, making it one of the largest in the CSU 
system. CSUN is comprised of 9 colleges, including the Eisner College of Education (COE), and 
the College of Science & Mathematics. These two colleges include 56 departments that offer 60 
baccalaureate degrees, 45 master’s degrees and 28 types of teaching credentials. In 2006-2007, 
the CSU system recommended 53% of the state’s teaching credentials (10,840), of which CSUN 
recommended 8% (844), the most of any CSU campus. 

The Institutional Context Salient to the SCALE Math and Science Partnership 

The factors listed below were identified as especially salient to the SCALE intervention, 
and then organized by the ICE categories. A plus or minus valence was assigned to each factor 
indicating whether, based on respondent views or our own interpretation of the relationship, it 
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supported or inhibited aehievement of SCALE goals. These factors are not static elements but 

interact with one another in a variety of ways. 

• External environment: Trends in educational reform take root at CSUN (+); budget crises 
frustrate faculty yet make external funding attractive (+/-); lack of pedagogical training in 
STEM doctoral programs (-); demand for STEM teachers in local K-12 districts (+); 
institution type (comprehensive) closely aligned with the Math and Science Partnership 
(MSP) goals (+); state credentialing policies discourage inter-disciplinary collaboration (-); 
lack of policies governing type and quality of professional development for K-12 teachers (- 
); and tense dynamics between IHE and K-12 educators (-). 

• Internal structure: Governance system emphasizes faculty autonomy (+/-); mission supports 
undergraduate education and teacher preparation (+); multi-tiered leadership generally favors 
pedagogical reform (+/-); limited pipeline of preservice science majors (-); some preservice 
programs require interdepartmental collaboration (+); large part-time workforce complicates 
efforts to reform instruction (+/-); recruitment, tenure, and promotion policies discourage 
teaching innovation (-); workload is heavy and not amenable to service (-); math department 
is strongly divided on pedagogical reform (+/-) existing reforms foster both support and 
skepticism (+/-). 

• Resources: Active and historic reforms fostered social network of STEM educators (+); 
extant K-12 professional development services (+); funding pressures limit resources (-); 
student body requires more remediation (-); status and social capital influence ability of 
faculty to participate in reforms (+/-). 

• Pervasive values: Tension between teaching mission and disciplinary traditions (+/-); 
scientific legitimacy and credibility equated with basic research (-); distinction between hard 
and soft science fuels inter-disciplinary tension (-); conflicting beliefs about the role of 
content and pedagogy in preservice STEM courses (+/-). 

• Individual sense-making: Eunding pressures and workload exert pressure on faculty time (+/- 
); faculty status influences time management (+/-); various personal reasons to participate in 
reforms (+); intellectual curiosity and willingness to take risks (+). 

• Practices: Some faculty maintain didactic approach to STEM instruction (-); some faculty 
experiment with inquiry-based approach (+); participation in interdisciplinary activities (+); 
participation in K-12/IHE collaborations (+). 

As the evaluation findings (below) indicate, these factors influenced how SCAEE was 

implemented. 



Evaluation Findings About SCALE Activities and Outcomes 

CSUN’s active involvement in SCALE began in spring 2005 as faculty began designing 
professional development (PD) institutes for K-12 teachers. Once introduced to CSUN, SCALE 
leaders from UW-Madison and CSUDH began interacting with a community of CSUN STEM 
and education faculty who were already active in the other reform initiatives. These faculty 
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became part of a larger cohort who were led by SCALE faculty and staff from UW-Madison and 
CSUDH. Apart from the CSLTN Provost, who had an advisory role in SCALE, no local CSUN 
staff participated in the SCALE leadership group. SCALE leaders primarily engaged CSUN 
STEM faculty and academic staff in science immersion institutes and math institutes, all of 
which were undertaken in collaboration with local K-12 districts. 

The science immersion institutes are a learning opportunity in which students are 
engaged in the scientific inquiry process over an extended period of time (4 weeks). Although 
described in some detail in the original SCALE proposal as science immersion experiences, no 
measurable objectives for IHEs were assigned to this activity, which makes it challenging to 
assess if and how SCALE achieved its goals in this area. 

Two factors within the institutional context of CSUN influenced the science immersion 
institute activity in different ways. A first factor was the demanding workload, which resulted in 
a limited pool of potential participants in the SCALE initiative. A key aspect of the workload is 
the high value placed on research productivity, which brings in valuable external funding in a 
time of budgetary crisis, elevates the prestige of individual faculty, their departments, and the 
entire university, and is an explicit goal of some CSUN administrators. As a result of this factor, 
some faculty indicated that it was not feasible or desirable to commit to an entire week of 
SCALE activities. A second factor that affected SCALE — and that somewhat mitigated the 
first — was the presence of a cohort of STEM and education faculty who were already engaged in 
K-12 PD activities. This cohort provided a ready-made pool of potential facilitators for the 
science institutes. 

The science immersion institutes were preceded by a design process that involved 
interinstitutional teams who collaboratively developed grade-specific science immersion units 
designed to address Los Angeles Unified School District (LAUSD) science standards. UW- 
Madison staff, CSUDH and CSUN STEM and education faculty, and LAUSD science experts 
and teachers collaborated to develop immersion units for fourth, sixth, seventh, and eighth 
grades. These cross-institutional teams then co-facilitated 5-day science institutes (five in 2006, 
eight in 2007) for a total of 270 LAUSD teacher participants. Anticipated (i.e., first-order) 
outcomes include providing professional development workshops for the LAUSD science 
teacher participants, engaging STEM faculty as learners and not solely as content experts, and 
providing pedagogical tools that STEM faculty (N=3) later used in their undergraduate courses. 
Unanticipated (i.e., second-order) outcomes include the diffusion of SCALE pedagogical 
techniques to other reform initiatives, expanding and training the STEM education cohort, and 
developing an intensive interaction with K-12 personnel. 

A series of math institutes (professional development workshops) for LAUSD middle and 
high school math teachers was the second primary SCALE activity at CSUN. These institutes 
were a joint effort of a CSUDH project funded by the Department of Education (QED) and 
SCALE. The leadership and materials for these institutes primarily came from CSUDH ’s 
mathematics department’s Center for Math and Science Education (CMSE). The goals of the 
math institutes were to increase student achievement in and understanding of the mathematics 
contained in the California state standards in Grades 6-9, and to better equip teachers to lead 
their students to a deeper understanding of mathematics. 
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The QED/SCALE math institute aetivity was influeneed by various factors in the 
institutional context of CSETN. One is the ideological schism that has long prevailed in the CSUN 
math department, which served as an important backdrop to the SCAEE math institutes, and 
effectively limited the pool of potential facilitators. The department is generally split into two 
camps with divergent perspectives on math education, one of which is, and one of which is not 
sympathetic towards a constructivist approach to math education. Indeed, one CSUN respondent 
noted that most education faculty only pay attention to members of the sympathetic group. The 
split in the math department played a direct role in the 2006 math institutes, as one of the 
facilitators proved to be somewhat antagonistic to the intent of math education reform in general, 
and the design of the math institutes in particular. Another factor influencing the math institutes 
was the Teachers for a New Era (TNE) initiative in which some math department faculty were 
already engaged prior to SCAEE, which led to information and ideas from one influencing the 
other. 



The math institutes were preceded by a design process that involved inter-institutional 
teams who collaboratively developed grade-specific math units designed to address EAUSD 
math standards. CSUDH and CSUN STEM faculty collaborated to develop math units for middle 
school (pre-algebra) and high school (algebra). Cross-institutional teams including CSUN STEM 
faculty, Madison Metropolitan School District (MMSD) personnel, and EAUSD personnel then 
co-facilitated 15-day math institutes (two in 2006, two in 2007) for a total of 83 EAUSD teacher 
participants. Anticipated (first-order) outcomes for this SCAEE activity include professional 
development workshops for the EAUSD teachers. Unanticipated outcomes (second-order) 
include complementing efforts of other reform initiatives (e.g., TNE), expanding and training the 
cohort of math educators, and becoming involved in the math department schism through a series 
of competing letters to the editor in the American Journal of Physics. 

Exploring How Cultural Phenomena Mediate and Influence STEM Education Reform 

Some pervasive values at CSUN posed a significant challenge to implementing the types 
of institutional and pedagogical changes envisioned by the SCALE project. To elucidate these 
challenges, we conducted a detailed, albeit exploratory, analysis of how cultural factors mediate 
and influence STEM education reform. Our reason for taking this fine-grained approach to 
studying culture is to articulate specific content and processes inside the black box called “CSUN 
cultural dynamics.” For this analysis, we employed a distributed theory of culture developed by 
cognitive anthropologists that is based on schema theory in cognitive science. Schemas are 
unconscious mental structures that distill information into generic units that are encoded into the 
neural networks of the brain through repetition (Brewer, 1987; D’Andrade, 1995). Researchers 
theorize that different schema combine to comprise mental models, which in turn collectively 
comprise the explanatory structures that actors use in a filtering process to omit or transform 
environmental stimuli (Johnson-Laird, 1983; D’Andrade, 1995; Strauss & Quinn, 1998). Some 
schema can be considered more cultural than others since they are internalized from instantiated 
cultural forms that are “part of the stock of shared cognitive resources of a community” (Shore, 
1996, p. 47). However, individuals differentially internalize these shared beliefs, values, and 
norms, and thus, communities are always “internally differentiated and cultural models are 
characterized by different degrees of sharing” (Shore, 1998, p. 1). An individual’s mental model 
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is then activated in response to environmental stimuli, which are largely composed of the unique 
context of the organization. 

Working within the interpretive tradition of Strauss & Quinn that uses natural discourse 
to identify cultural models (1998), we analyzed the mental model for STEM education reform of 
one faculty member as a way to explore the relationships (for this person) among institutional 
context, culture, individual cognition, and STEM education reform. This respondent’s mental 
model was largely comprised of four cultural schema pertaining to STEM education reform that 
we found to be common across social groups and administrative units at CSUN. These cultural 
schema include: 

• Scientific legitimacy and credibility equated with basic research accomplishments; 

• Distinction between hard and soft sciences; 

• Tension between institutional support for reform and the disciplines; and 

• Divergent beliefs about the relative importance of content and pedagogy in pre-service 
STEM courses. 

It is important to note that we do not claim that all faculty, administrators, and staff at 
CSUN exhibit these values or beliefs. Instead, while these values were expressed by several 
respondents across different social groups and administrative units, individuals may 
differentially internalize and exhibit these cultural schema depending on their personal 
backgrounds, personality, social position, discipline, and many other situational or demographic 
variables. In addition, these cultural schema are related to specific policies, institutional 
structures, and other contextual factors. Anthropologists sometimes describe clusters of similar 
practices (e.g., rituals, tool-making, languages) in a bounded geographic area as “cultural 
complexes.” Eollowing this usage, we consider these schema pertaining to STEM education 
reform and their related contextual determinants to be a cultural complex for scientific legitimacy 
and credibility. We believe that this STEM education reform cultural complex is particularly 
resilient for individuals, in part because it is developed through years of intense training and 
socialization in communities of practice whose membership is carefully screened (Lave & 
Wenger, 1991). 

This particular respondent did not participate in SCALE, and was hired relatively 
recently at CSUN — ^the respondent’s first academic appointment. The primary contextual factor 
that shaped the respondent’s mental model for STEM education reform was that of workload and 
attendant reward, tenure, and promotion considerations. As is common for junior faculty, this 
respondent was singularly focused on conducting research, publishing research articles, raising 
external research funds, and teaching the obligatory 12 units per semester. These immediate and 
practical concerns about workload were related to the cultural schema about scientific legitimacy 
and credibility. Since participating in a reform effort would likely impede progress towards 
tenure and promotion, this person turned down offers from SCALE. Concern about tenure also 
made approval from departmental peers particularly important, which led this person to place 
high value on social factors in the college and department. Of note, the cultural schema 
comprising this person’s mental model were expressed as taken-for-granted aspects of academic 
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life in a STEM discipline. As a doctoral student in a major research university, the respondent 
had never been in an institutional environment where the prioritization of research over teaching 
and service activities was not the case. Furthermore, these schema were reinforced locally at 
CSUN by the department’s reward, tenure, and promotion (RTF) policies and CSUN leadership. 
Indeed, some CSUN leaders had conveyed explicit messages regarding the expectations for 
faculty research productivity, including the statement at a public meeting that junior STEM 
faculty “should be living in the lab.” 

Since participating in a reform effort would likely impede progress towards tenure and 
promotion, this person did not participate in any reform efforts, including SCAEE. However, 
since the TNE effort did engage the department in an attempt to hire a pedagogy specialist, this 
respondent’s deep-seated cultural schema related to STEM education was engaged. Through this 
process, this respondent’s (and colleagues’) deep-seated cultural schema related to STEM 
education was engaged. Since research was considered the top priority in the department, the 
respondent felt that while the TNE position “would be nice,” it would essentially be useless 
since, “(I)nvesting time in becoming a good teacher is not rewarded and we are not really 
incentivized to spend inordinate amounts of time developing new teaching tools.” Thus, the 
cultural schema that equates scientific credibility and legitimacy with basic research, and its 
reinforcement in RTF policy, rendered this reform attempt superfluous and actively at odds with 
the prevailing value system of the department. Ultimately, this faculty’s mental model did not 
change as a result of the SCAEE project and other reform efforts. In fact, the failure of the TNE 
effort to hire a pedagogy specialist may have reinforced their mental model by demonstrating 
that a research-centered perspective is truly the dominant one in the department, in spite of the 
allure of a fully funded tenure-track appointment. 

Conclusions 

Without measurable objectives with which to evaluate the SCAEE project’s activities and 
progress, it is difficult to make a definitive statement about the relative success or failure of the 
project. However, it is fair to say that in light of the original goal to affect changes to 
undergraduate STEM instruction, interdisciplinary collaboration on preservice programs, 
interinstitutional collaborations on in-service programs, and the underlying institutional culture 
that informs these activities, the effects of the SCALE project must be considered modest. Of 
note, SCALE focused its efforts on a relatively small and discrete point within the CSUN 
context — engaging faculty to support K-12 teacher professional development. It is also fair to 
say that the SCALE project resulted in outcomes pertaining to the institutional context of CSUN 
that directly and indirectly relate to the four goals of the project. This was possible largely 
because the SCALE initiative dovetailed with other extant reform initiatives that collectively 
created a very favorable environment for STEM education reform. 

We conclude that there was an inconsistency between the goals SCALE leaders stated 
and the implementation of these goals. Since no SCALE leaders were present at CSUN, there 
was no explicit theory of institutional change for CSUN, in contrast to SCALE activities at both 
UW-Madison and CSUDH. Instead, SCALE engaged CSUN as primarily a site for its already 
existing math and science institutes, and largely ceded the effort at institutional transformation to 
chance and the extant reforms at CSUN. Since the SCALE leaders at UW-Madison and CSUDH 
did not have local expert knowledge, the campaign approach to change used at the other SCALE 
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IHEs was not evident at CSUN. Instead, SCALE in effeet planted its interventions at CSUN and 
relied on loeal experts to eonduet the projeet. An enduring lesson from SCALE at CSUN is that 
multi-institutional reform efforts should focus on recruiting local leaders who can be sufficiently 
informed about and active at each partner institution. One possible effect of this approach was a 
lost opportunity to coordinate SCALE with other extant STEM education reform initiatives. 

Other core findings are derived from the analysis of the situated mental model of the 
CSUN faculty, and of the pervasive values and beliefs at CSUN pertaining to STEM education 
reform. 

1 . There is a cultural complex of scientific legitimacy and credibility. 

2. Mental models are hard to change. 

3. Contextual factors shape mental models. 

4. Professional communities are critical in culture formation. 

5. A comprehensive approach to culture change and culture-brokers is important. 

Taken as a whole, this analysis suggests that shaping the culture of an organization may 
require comprehensive efforts to change the structural, social, and symbolic milieu in which 
individuals operate, in addition to efforts to change the cognitive processes that constitute 
individuals’ habits of mind. Such efforts may require leaders to employ a flexible and 
multifaceted toolkit of frames through which to analyze their organizations (see, for example, 
Bolman & Deal, 2003). In addition, such an effort would require campus and other reform 
leaders to take a deliberate and strategic approach, as opposed to pursuing an intervention that 
unfolds according to chance. It is especially important that a local figure play a leadership role, 
as mental models within an organization may operate according to a logic that is inaccessible to 
external leaders and change agents. In cases where interinstitutional or interdisciplinary 
collaborations are taking place, it may also be important for a local individual to play the role of 
a culture broker. Our case studies of UW-Madison and CSUDH indicate the importance of such 
individuals, who understand the cultural schema that operate within different groups, and can 
carefully negotiate the divisions or tensions that may exist between the groups and their 
pervasive values and beliefs. 

We also suggest that, of the many contextual factors identified in this study, the following 
stand out as especially important for leveraging the types of change sought by the MSP program; 

1 . Points of contact with the external environment; 

2. Leadership at all levels; 

3. Decision-making bodies and interdepartmental forums; 

4. Networks of STEM educators; 

5. Other reforms; 
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6. Cultural complex regarding diseiplinary legitimacy and credibility; and 

7. Individual faeulty workloads 

We suggest that the above factors deserve partieular attention by eampus leaders and 
STEM edueation reformers beeause they have the potential to affeet a cascade of impacts in a 
variety of points in the institutional eontext of an IHE. In so saying, we eaution that these seven, 
as well as other leverage points should not be viewed as isolated magie bullets that ean produce 
fast and enduring reform across the entire university. Rather, they must be understood as 
operating in dynamie interaction, such that a change in one may yield unpredietable and even 
impereeptible movement in others. 

We also reeommend that future STEM education reformers at research universities, and 
poliey makers at funding ageneies such as the NSE and the Department of Education may benefit 
if they: 

• Conduct assessments of the institutional context prior to program planning and 
implementation; 

• Ensure that reeruitment efforts pay attention to workload and cultural factors; 

• Design neutral spaees in whieh different groups may interaet; 

• Reeruit a skilled culture-broker when working with interdisciplinary groups; 

• Marshal existing resourees and reform projeets to collectively target key leverage points; 

• Focus on developing eohorts of STEM edueators in speeifie departments; and 

• Carefully design top-down structural reforms with attention to the moving parts of the 
institution 

Potential benefits of this research to the STEM education community are twofold: (a) this 
approaeh may enable evaluators to systematically assess if and how an institution’s eontext and 
culture aet as mediating variables that infiuenee reform, and as outcome measures for programs 
aiming to affect institutional transformation or cultural change; and (b) given the widespread 
interest in changing the eulture of schools and IHEs, such insights would give program planners 
and poliey makers the tools to better understand their institutions and the ultimate effects of 
investments in reform. 
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A Final Case Study of SCALE Activities at California State University, 
Northridge: How Institutional Context Influenced a K-20 STEM Education 

Change Initiative 

Matthew T, Hora and Susan B, Millar^ 

Institutions of higher education (IHE) play an important role in mathematics and science 
education by providing undergraduate instruction, operating teacher training programs, and 
providing in-service training for K-12 teachers. The National Science Foundation (NSF)-funded 
System- Wide Change for All Feamers (SCAFF) project sought to effect change in its partner 
IHFs by: (a) improving science, technology, engineering, and mathematics (STEM) 
undergraduate education; (b) improving collaborations between STEM and education faculty 
regarding preservice programs; (c) improving collaborations between IHE faculty and K-12 
districts regarding in-service training; and (d) improving the institutional policies and practices 
that support these activities. As part of the SCAFE IHE case studies line of work, this document 
provides findings on the effects of the SCAFE project, at the California State University, 
Northridge (CSUN) between March 2005 and August 2007. This case study includes two 
interrelated accounts of SCAFE activities: (a) presentation of evaluation findings for each of the 
SCAFE activities undertaken at CSUN, and (b) analysis of how specific aspects of the 
institutional context influenced SCAFE activities. This case study is part of a larger effort to 
develop and field test an evaluation design that strives to measure how an educational 
organization’s unique contextual features interact with an intervention. This larger effort includes 
case studies of the University of Wisconsin-Madison (UW-Madison) and California State 
University, Dominguez Hills (CSUDH) (Hora & Millar, 2007; Hora & Millar, 2008). 

The NS F Mathematics and Science Partnerships Program 
The Problem: Declining Performance of U.S. Students in Mathematics and Science 

The performance of U.S. students in mathematics and science has become an increasingly 
pressing problem, particularly in light of the implications for the future competitiveness and 
employability of U.S. residents. As numerous studies and reports attest, the problem is systemic, 
with challenges including public policy, funding, and curricular strategies that span the 
educational continuum from higher education to K-12 (Committee on Science, Engineering, and 
Public Policy [COSEPUP], 2006; National Research Council Committee on Science and 
Mathematics Teacher Preparation [CSMTP], 2000; Project Kaleidoscope, 2006; U.S. 

Department of Education, 2006b; U.S. Office of Science and Technology Policy, 2006). Most 
recently, researchers and policymakers are focusing on the importance of a teacher workforce 
that is more highly trained in science and mathematics (Eevine, 2006; U.S. Department of 
Education, 2005). Indeed, the 2006 COSEPUP report suggests that an appropriate goal to address 
the eroding U.S. advantages in mathematics and science is to produce 10,000 qualified teachers 
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annually. This goal addresses the “ehronie and growing shortage of discipline qualified K-12 
teachers” that researchers have been warning policymakers about for several years (Seymour, 
2001). This shortage is illustrated by the fact that in 2000, 93% of students in Grades 5-9 were 
taught physical science by an instructor who lacked a college major or certification in the 
physical sciences (National Center for Education Statistics, 2004). The mandate of the No Child 
Left Behind Act (NCLB; 2002) that all school districts must employ only “highly qualified 
teachers” further indicates that the issue of teacher workforce quality in science and mathematics 
is a critical national issue. 

One of the many challenges in reforming teacher preparation and professional 
development practices in the U.S. is the complex nature of the preparation process. For example, 
in order to qualify for certification to teach at the K-12 level, most future mathematics and 
science teachers must navigate both teacher preparation programs in schools of education, and 
disciplinary requirements in STEM departments at accredited IHEs. Then, they participate in 
professional development programs that are governed by state and/or district policies, and 
offered by an array of providers including private vendors, district specialists, and IHE faculty. 
Thus, individual K-12 teachers obtain their mathematics and science content and pedagogical 
training from diverse institutions and stakeholders whose programs are governed by diverse 
policies that operate in isolation and with little coordination. As a consequence, the quality of 
this training often is uneven, if not haphazard (Mundry, Spector, Stiles, & Loucks-Horsley, 
1999). In 1998, the NRC addressed this multi-institutional problem by establishing a Committee 
on Science and Mathematics Teacher Preparation (CSMTP). The CSMTP report (NRC, 2000) 
states that a significant restructuring of the relationship between K-12 schooling and higher 
education, including new partnerships to collaboratively design and implement high-quality 
professional development programs, is required to adequately prepare and train effective 
teachers. 

The NSF’s Math and Science Partnership Program 

Growing concerns about improving the alignment of the teacher training continuum are 
among the reasons the NSF has invested substantially in teaching improvement and 
organizational change in higher education — most recently through its MSP program. These 
concerns reflect development in some national policy makers’ understanding of the role that 
higher education plays in preparing future teachers, moving beyond long-held critiques of 
teacher preparation programs to a closer examination of the role of disciplinary faculty in the 
STEM disciplines. 

The NSF MSP program aims to improve the coordination among STEM undergraduate 
education, teacher preparation programs, and K-12 professional development by fostering 
mutually beneficial partnerships between IHEs and K-12. Specifically, it hopes to encourage 
partnerships between STEM disciplinary faculty, education faculty, and IHE administrators with 
the K-12 districts they serve in “efforts to effect deep, lasting improvement in K-12 
mathematics and science education” (NSF, 2002). The MSPs are based on the premise that 
IHE/K-12 partnerships should draw on the disciplinary expertise of STEM faculty and graduate 
students, and undergraduate STEM (including preservice) students to develop strong math and 
science content knowledge and pedagogical methods. That is, the MSP program’s theory of 
change includes the idea that increased involvement of STEM faculty in the teacher training 
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continuum will result in lasting improvements in K-12 student learning (CASHE, 2006; NSF, 

2002). 

Specific Problems Addressed by the MSP Program 

STEM undergraduate instruction. Critiques of the quality of teaching in higher 
education began in the 1980s with A Nation at Risk, by the National Commission on Excellence 
in Education (1983). Since then, we have seen a cascade of critieisms of higher education, 
culminating in the U.S. Department of Edueation’s A Test of Leadership (2006). Critics note that 
many STEM undergraduate majors graduate with substantial deficiencies in their content 
knowledge (e.g., Handelsman et ah, 2006). Researchers have identified high rates of attrition 
among undergraduate seience majors as one of the consequences of poor undergraduate 
instruction and academic assistance (Seymour & Hewitt, 1997). Beeause in most states students 
seeking to earn secondary school teaching credentials are among these science majors, and in all 
states students seeking to earn primary and secondary school teaching credentials take STEM 
eourses, national poliey makers are increasingly reeognizing and serutinizing the roles that 
STEM faeulty play in the teacher training continuum by instructing preservice eandidates in 
disciplinary content and modeling pedagogical methods. Eor example, the Shaping the Future 
report by the NSE (1996) recognized these roles when it urged STEM faculty to use active 
learning strategies in their undergraduate courses not only to help students understand diseipline 
eontent more deeply but also to model effeetive pedagogy for future teaehers. 

Teacher preparation programs. Teacher preparation programs and the colleges of 
education that operate them have been subject to criticism for years. In particular, critics charge 
that college of edueation curricula for preservice eandidates is poorly designed and insuffieiently 
grounded in rigorous content courses and/or pedagogical instruction (Eabaree, 2006; Mundry et 
ah, 1999). Policy bodies such as the CSMTP (NRC, 2000) and NSF-funded practitioner 
reformers (Millar & Alexander, 1996) urge greater collaboration across departments and colleges 
within an IHE with respeet to teaeher preparation. In response to these eritiques and 
recommendations, many initiatives both within and outside IHEs are underway to improve how 
teachers are prepared and trained (Robinson, 2006). Among these initiatives are several — 
ineluding the NSF’s Collaboratives for Exeellence in Teaeher Preparation program and the MSP 
program — that foeus on the role of STEM and edueation faeulty in organizing and delivering a 
solid curriculum. However, critical gaps remain in our understanding of teacher education 
program effectiveness, including the effects of subject-matter coursework on teacher knowledge 
(Cochran-Smith & Zeiehner, 2005), and the relative effieaey of different teacher edueation 
pathways (Darling-Hammond, Chung, & Frelow, 2002). 

IHE participation in professional development programs. In-service training in 
disciplinary content and pedagogical methods, which authorities suggest should occur on a 
regular basis (U.S. Department of Education, 2005), is another key venue for enhancing K-12 
teacher mathematics and science knowledge. There is a large body of researeh on the effieaey of 
professional development programs, and researchers are increasingly questioning the efficacy of 
the traditional model of professional development, where IHE faculty or other experts deliver 
knowledge to K-12 teachers (Caret, Porter, Desimone, Birman, & Yoon, 2001). This approach is 
considered ineffectual because it is decontextualized, treats teaching as a routinized and technical 
activity, and stresses “additive rather than transformative change” (Carlone & Webb, 2006, p. 
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545). Possible solutions to this problem inelude paying eloser attention to the eontext of 
professional development design (Ball & Wileox, 1989), fusing eontent and pedagogy by 
involving both diseiplinary and edueation IHE faculty (U.S. Department of Education, 2005), 
and more explicitly building on novice teacher’s prior experiences or knowledge (Mundry et ah, 
1999). 



Challenges to higher education reform and institutional change. The MSP program is 
facing the extremely difficult undertaking of fostering change in higher education, a sector 
known to be very resistant to change (Cuban, 2000). Researchers cite the persistence and 
resilience of institutional tradition (Kezar & Eckel, 2002), the decentralized and “loosely 
coupled” nature of IHEs (Birnbaum, 1988), and the unique elements of organizational structures 
and autonomous cultures as characteristics of IHEs that make them resistant to change efforts. 
Eurthermore, historic divisions between STEM and education faculty, and between higher 
education and K-12 education, may inhibit collaborative activities between the two sectors 
(Labaree, 2006; Gilroy, 2003). These challenges are pertinent to the MSP program, and may 
account for limited effects of this program on STEM faculty and institutional processes. Eor 
example, a 2006 review of institutional changes of 21 MSP higher education partners found that 
curricular changes are occurring at IHEs across the MSPs, but with a majority of the changes in 
preservice programs and in-service professional development, and not in STEM departments. 
Eurthermore, changes were at the individual level instead of the institutional level, with no 
department-wide initiatives or collaborative team efforts (CASHE, 2006). An analysis of STEM 
faculty engagement in the MSP program similarly found little evidence of institutional change, 
but significant individual-level shifts in STEM faculty knowledge of and participation with K-12 
education (Zhang et ah, 2007). The Zhang et al. study also found that the effect of STEM faculty 
engagement in the teacher training continuum is difficult to ascertain, and that effects on student 
learning are even more elusive. 

SCALE Theory of Change and Goals Regarding IHEs 

The MSP project featured here, SCALE, sought to effect change in its partner IHEs by 
creating a transformative culture in IHEs through the creation of cross-cultural working teams 
who operated at the intersections among K-12 districts, colleges of education, and colleges of 
mathematics, science, and engineering (SCALE, 2005). Upon engaging the IHE case studies 
team, they conveyed their theory of action regarding IHEs by stating that SCALE seeks to 
achieve the following goals: 

1 . Reform undergraduate STEM courses; 

2. Promote collaboration between STEM and education departments on preservice teacher 
education; 

3. Promote collaboration between IHEs and K-12 districts on in-service professional 
development; and 

4. Improve institutional policies and practices at the IHE level that support faculty engaged in 
pre- and in-service activities. 
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However, SCALE leaders did not define or operationalize the eonstruet of organizational 
eulture or state measurable objeetives for the four goal areas artieulated for the IHE ease studies 
team. Unable to measure progress towards a set of elearly defined objeetives, or evaluate the 
program aceording to a set of established eriteria, this evaluation design focused on describing 
program activities, assessing how well subsequently observed effects met stated goals, and 
analyzing the relationship between the institutional context (with a specific focus on cultural 
phenomenon) and program activities. Given the paucity of research designs and research-based 
indicators for measuring institutional change over time, conducting such an evaluation became 
problematic. Thus, this effort is in many ways an exploratory effort to develop and field test an 
evaluation design that measures how an educational organization’s unique contextual features 
interact with an intervention. 



Methodology of the IHE Case Studies 

This section includes a brief description of the rationale for the research, the theoretical 
framework guiding the research, and the research design. For a more detailed description of the 
study’s methodology, see the Appendix. 

Conceptual Framework 

This evaluation is designed and implemented with careful attention to the need for high- 
quality evaluation of STEM education programs, particularly in complex institutional 
environments such as IHEs. To achieve the goal of improving K-20 STEM education, policy 
makers, funders, and practitioners are placing greater weight on methodological issues in 
program evaluation, with particular focus on basing decisions about program replicability and 
expansion on knowledge-based claims (Eawrenz & Huffman, 2006; Mosteller, Nave, & Miech, 
2004; Kelly & Yin, 2007). Programs such as the NSF MSP program entail “research and 
development, requiring high quality evidence of effectiveness in order to ensure intellectual rigor 
and broad applicability” (NSF, 2005). These requirements further contribute to an interest in 
evaluation.^ There is also widespread pressure to improve the methodological rigor of 
evaluations of STEM education programs (U.S. Department of Education, 2007). Some 
researchers note that education research has had a difficult time establishing itself as a science, in 
part due to the lack of accumulated knowledge, replicable studies, and transparency about 
methodological issues (Kelly & Yin, 2007). 

Studying and evaluating change processes in complex institutional environments presents 
two inter-related challenges that are addressed in this qualitative case study. First, researchers 
who study STEM education reform are surfacing difficulties associated with studying change 
processes in complex institutional environments (Patton, 2006; Clune et ah, 1997). A particular 
challenge found in many evaluation studies is that researchers have tended to focus on discrete 
elements, such as curriculum or assessment practices. For example, Anderson and Helms (2001, 
p. 4) argue that reformers need to “come to grips with the totality of this complex situation” in 



^ The MSP program sponsored a 2008 eonferenee with the theme “Claims-Based Outeomes: What do we know? 
How do we know what we know? What do we still need to know?” 
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order to fully understand these eomplex dynamie systems, and if and how they should aeeept, 
reject, or adapt reform initiatives. 

Second, research on reform implementation in both higher education and K-12 has also 
found that policy directives are frequently adapted and transformed by individual agents at the 
local school or IHE level (Ball & Wilcox, 1998; Spillane, Reiser, & Reimer, 2002). The process 
of interpreting policy interventions and adapting them to one’s own local situation is sometimes 
called sense-making, where institutional actors “make sense” of their environment and select 
appropriate actions (Birnbaum, 1988). Increasingly, researchers are also recognizing the 
importance of this process and of how local contextual factors such as organizational structure 
and leadership shape the sense-making process, which is an important departure from the view 
that teachers operate in classrooms as if they are immune from exogenous forces such as the 
MSP program (Coburn, 2001; Spillane, Reiser, & Reimer, 2002). These topics are interrelated 
because local contextual factors shape the sense-making process, as individuals may be inclined 
— if not required — to attend to contextual features of their organizations, especially the cultural 
norms of administrative units such as academic departments (Gamoran et ah, 2003; Cobum, 
2001). However, research in this area is complicated by a paucity of theories and methods that 
adequately address the multilevel and recursive nature of the relationships among context, 
culture, and individual cognition, particularly for application in program evaluations. 

In response to these concerns, this study of SCALE at CSUN seeks to identify contextual 
factors salient to STEM education reform, their relationship to group- and individual- level 
cultural phenomenon (i.e., mental models), and their collective influence on the SCALE 
intervention. Our methodological approach is to analyze the SCALE project through the analytic 
lenses of institutional theory and cultural models theory from cognitive anthropology. Thus, we 
are treating IHEs as complex organizations comprised of technical systems, political entities, 
social and cultural communities, and individual-level practices, each of which interact and 
collectively comprise the institution’s context (DiMaggio & Powell, 1983; Colbeck, 2002). In 
addition, an institution’s context is not a passive backdrop to individual actions, but is itself a 
dynamic setting in which individual agents are located according to the interactions among their 
own habitus (i.e., personal dispositions), capital (social, economic, or cultural), and the mles of a 
particular field (Bourdieu, 1977). The institutional field also extends beyond a single 
organization’s boundaries to include clusters of similar institutions (e.g., comprehensive IHEs), 
and the “social conditions under which inter- and intraorganizational power relations are 
produced, reproduced, and contested” (Emirbayer & Johnson, 2008, p. 1). 

In addition, we employ here a distributed theory of culture in contrast to a commonly 
used theory of culture that seeks to establish a unitary and stable set of cultural norms for a 
specific group of people (e.g., the math department). This approach focuses on the distribution of 
knowledge and beliefs across and within groups, with particular attention to the differential 
internalization of cultural norms by individual actors based on an individual’s cognitive 
processes and unique position in the local social, political, and cultural environment (Shore, 

1996; Atran, Medin, & Ross, 2005). The underlying research paradigm guiding this approach is 
that of ethnographic research, where we attempt to describe the IHE context and the SCALE 
implementation in a grounded and multidimensional fashion, based largely on the perspectives 
and experiences of local participants (Agar, 1996). Potential benefits to the STEM education 
community are twofold. Eirst, for programs aiming to affect institutional transformation or 
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cultural change, this approach may enable evaluators to systematically assess if and how an 
institution’s context and culture act as (a) mediating variables that influence reform, and (b) as 
specific outcomes of the reform itself. Second, given the widespread interest in changing the 
eulture of schools and IHEs, the approaeh may offer program planners and poliey makers tools to 
better understand their institutions and the ultimate effects of investments in reform. 

Research Design and Methods 

The research questions for the IHE ease studies line of work are informed by the dual 
need to evaluate the SCALE MSP and to more deeply examine the reasons why SCALE did or 
did not achieve its goals. Hence, we posed these researeh questions — which mirror the SCALE 
theory of ehange — about eaeh IHE studied. 

1 . How does the institutional context influenee STEM instruction, STEM and education faculty 
collaborations on preservice programs, and IHE and K-I2 collaborations on in-service 
programs? 

2. Are SCALE activities eontributing to ehanges in SCALE’S primary goal areas? If so, how? 

3. Under what eonditions are change initiatives, including SCALE, accepted and incorporated 
at the institution? 

This case study uses a repeated cross-sectional design, with in-depth interview data 
eollected at Time 1 (Tl; June 2006) and Time 2 (T2; June 2007). Other data collected for this 
research included official university and SCALE doeuments, and observations of SCALE 
meetings. Non-random sampling proeedures were used to identify interview respondents, which 
included 18 at Tl and 16 at T2, for a total of 34 interviews with 25 unique individuals. In 
addition, we draw on interviews condueted with SCALE leaders and UW staff, eondueted for the 
CSUDH and UW-Madison case studies to colleet information on program planning, 
implementation, and evaluation. 

Interviews were coded using a structured approach to grounded theory which is based on 
a eoding scheme we eall the institutional context framework (ICE). The ICE was developed as 
part of the preliminary analyses of the SCALE IHEs (Hora & Millar, 2006), and includes the 
following code families: external environment, internal structure, resources (fiscal and social), 
pervasive values, individual sense-making, and praetiees. The ICE is our eontribution towards 
systematieally identifying aspects of the institutional eontext (i.e., indicators) that influence 
STEM education reform. Using the ICE, interview data were systematically coded, reduced, and 
inductively analyzed for reeurring patterns and themes in order to identify contextual factors 
salient to SCALE, group- and individual- level eultural norms, and individual sense-making 
proeesses. Generally, at least three observations regarding a specific aspect of the institutional 
context (e.g., tenure policies) or SCALE activities were required for inclusion in the analysis. 
Additional analytic procedures included (a) a causal network analysis that allows for the graphic 
representation of the relationships among these multilevel phenomenon, and (b) an exploratory 
analysis of mental models and their eonstituent eultural schema, as they are situated within the 
institutional context of CSUN. We used established methods of qualitative analysis to verify our 
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findings, including triangulation of sources, actively seeking diseonfirming evidenee, and 
member-eheeking findings to ensure their aecuraey. 

Limitations 

This research is designed to (a) explore faeulty sentiments at one intervention site, (b) 
investigate the initial impaet of SCALE aetivities at that site, and (e) generate a theoretieal and 
practieal approaeh for analyzing STEM edueation projeets. Aeeordingly, the sample of 
respondents interviewed for this researeh does not eonstitute a random or representative sample 
of CSUN overall, or of individual CSUN eolleges or aeademie departments, and is not intended 
to be generalizable to other IHEs or other IHE faeulty. This micro-level analysis builds on the 
strength of the ethnographie ease study approaeh. The interpretations and elaims in this ease 
study go only as far as is warranted by the methods used and the data eolleeted. Sinee the 
preliminary IHE ease studies were also intended to provide feedback for SCALE administrators 
and practitioners, it is possible that these ease studies influeneed the outeomes of SCALE and the 
findings reported here. Einally, attrition of faculty and program participants at CSUN resulted in 
different populations available for interviews at T1 and T2. As a result, reported ehanges are 
based on data from a variety of respondents at both points in time, and do not represent the 
observations or experienees of a single eohort over time. 

II, A Snapshot of the Intervention Site: CSU Northridge 

In this seetion we briefiy review the features that are prominent elements of CSUN, the 
field in whieh the intervention was enaeted. A more detailed analysis of how these and other 
features dynamieally interact and actually serve as mediating variables for the SCALE 
intervention is provided later in this seetion. 



Mission 

Part of the California State University (CSU) system, CSUN focuses on undergraduate 
edueation and provides affordable opportunities for a broad eross-seetion of California’s 
population (CSU System, 2008). IHE mission statements serve as publie deelarations of an 
institution’s primary goals and objeetives; CSUN’s mission statement follows: 

California State University, Northridge exists to enable students to realize their 
edueational goals. The University’s first priority is to promote the welfare and intelleetual 
progress of students. To fulfill this mission, we design programs and aetivities to help 
students develop the aeademie eompeteneies, professional skills, critieal and ereative 
abilities, and ethieal values of learned persons who live in a demoeratie soeiety, an 
interdependent world, and a teehnologieal age; we seek to foster a rigorous and 
eontemporary understanding of the liberal arts, seienees, and professional diseiplines, and 
we believe in the following values. (CSUN, 2008) 

In addition, CSUN’s stated values inelude a eommitment to teaehing, seholarship, and 
aetive learning, a eommitment to exeellenee, respeet for all people, and allianees with the 
eommunity. 
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Demographics, Academic Programs, and Governance 



Institution Type 

CSUN is in the middle tier of California’s three-tiered system of higher edueation. This 
three-tiered system is eomprised of the researeh universities in the University of California 
system, the master’s-granting universities in the California State University system, and the 
eommunity eollege system. In terms of the influential national Carnegie Foundation ranking of 
IHEs, CSUN is a “Master’s-L” (also known as eomprehensive), a designation for large 
universities that do not grant doeto rates, but have master’s level programs (Carnegie Foundation 
for the Advaneement of Teaehing, 2006). The aecrediting ageney for CSUN is the Western 
Association of Schools and Colleges. 

Student Body 

The demographics of the CSUN student body (both undergraduate and graduate students) 
in the fall of 2007 were as follows: 32% white, 14% other, 27% Latino/a, 12% Asian-American, 
8% African-American, 6% International, and under 1% American Indian and Pacific Islanders 
(CSUN Office of Institutional Research, 2008). The fall 2007 enrollment at CSUN was 35,446, 
which makes it one of the largest of the 23 California State Universities (only smaller than the 
CSU institutions at Fullerton, Long Beach, and San Diego). Of the total enrollment, only 77% 
(27,418) of the students were full time, a fact that places CSUN at a disadvantage relative to 
CSUs with a higher proportion of full-time students, because state funding policy is based on 
“full-time equivalents” rather than “headcounts.” The CSUN student body is 88% undergraduate 
and 12% graduate students, 59% female and 41% male. With the average age of undergraduate 
and graduate students at 23.3 years and 33.5 years, respectively, many students are not traditional 
age. The numbers of first time freshmen (4,130) and new undergraduate transfers (3,989) are 
almost identical, and 83% of these transfers come from California community colleges (CSUN 
Office of Institutional Research, 2008). The CSUN student body, as some respondents noted, 
differs significantly from that of research universities, where most students matriculate directly 
from high school, live on or near campus, and graduate within four or five years. 

Degree Programs at CSUN 

In the 2006-2007 academic year, CSUN conferred 5,682 bachelor’s degrees and 1,623 
master’s degrees. In fall 2007, the top undergraduate majors were general psychology (2,068 
students), liberal studies (1,800), radio/TV broadcasting (1,466), business administration (1,459), 
sociology (1,322), and organizational systems management (1,297). The liberal studies program 
is a popular option for students intending to become elementary school teachers. Biology (1,099 
students) is the only STEM discipline in the top 10 majors at CSUN. Education degrees 
dominate the master’s level: the top graduate degrees that year were educational administration 
(627), counseling (403), and special education (279). (CSUN Office of Institutional Research, 
2008). 
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Organizational Structure and Governance 

CSUN is comprised of nine eolleges, ineluding the Eisner College of Edueation and the 
College of Seienee & Mathematies. There are 56 departments within these eolleges, whieh offer 
60 baeealaureate degrees, 45 master’s degrees and 28 types of teaehing credentials. There are 
also numerous researeh eenters and institutes at CSUN, ineluding the Center for Researeh and 
Innovation in Elementary Edueation, and the Inter-diseiplinary Mathematies based Research and 
Education Center. The administration is eomprised of a president, a provost (who is also viee 
president for aeademie affairs), and viee presidents for student affairs, administration and 
finanee, and university advaneement (CSUN, 2006). The offiee that is most direetly engaged in 
SCAEE-related aetivities is the provost and aeademie affairs. The governanee system in the CSU 
system emphasizes departmental and faeulty autonomy regarding the edueational funetions of 
the university. 



Faculty and Workforce Characteristics 



Instructional Workforce 

Of the 2,159 faeulty at CSUN in fall 2007, 1,308 were part time (61%), and 851 (39% 
were full time (CSUN, 2008). Some of the part time faeulty are what respondents ealled 
“freeway fliers:” instruetors who eommute on Los Angeles freeways from eampus to eampus 
sinee they hold temporary appointments at multiple IHEs in the area. The eomposition of 
CSUN’s workforee is eonsistent with national trends in higher edueation. 

Faculty Workload 

Aeeording to CSUN doeuments, the faeulty workload at CSUN is governed by two 
polieies, the California Eaeulty Assoeiation (CEA) eolleetive bargaining agreement (CSU, 2006) 
and the CSUN Administrative Manual, Seetion 600 Aeademie Personnel Polieies and Procedures 
(CSUN, 2005). The CPA agreement sets forth the broad parameters of faeulty workload, 
benefits, and personnel polieies that the CSU system must observe (CSU, 2008). The CSUN 
administrative manual further speeifies that the normal faeulty instruetional load is 12 weighted 
units of instruetion and the equivalent of 3 weighted units for advisement, eommittee 
assignments, and offiee hours. However, administrators may reduee this instruetional load by 
assigning non-teaehing responsibilities sueh as administrative duties, instruetion related 
aetivities, researeh or program development aetivities reimbursed by the university or external 
funders, and leaves of absences without pay (CSUN, 2006). Thus, each faculty member at CSUN 
is generally expeeted to teaeh the equivalent of four eourses a semester, and satisfy 
administrative duties and student support serviees. 

Recruitment, Tenure, and Promotion Policies 

The faeulty reward and promotion system, also known as RTP, is one of the most 
important and influential aspeets of institutional life. At CSUN, these polieies are governed by 
statutes in the Administrative Manual, Seetion 600: Aeademie Personnel Polieies & Proeedures. 
Seetion 621.1 of the manual states that in making appointments, the following factors should be 



10 



A Final Case Study of SCALE Activities at CSUN 



considered: exeellence in scholarship and training, interest and skill in teaehing, promise of 
professional growth, and qualifications of personal maturity (CSUN, 2005). For tenure review, 
the procedure is as follows: 

1 . A department personnel committee reviews personnel files and submits a recommendation. 

2. The department ehair makes an independent evaluation, 

3. The personnel committee of the college, the dean, and finally the president submit 
rec ommendations . 

4 The president makes the final deeision. 

In reviewing offieial CSUN documents, we found evidenee that the policies governing 
RTF at CSUN encourage and support the practice of active learning pedagogies and research 
activities on teaching and learning. The university-wide RTF guidelines specifically identify 
pedagogy-based researeh, as well as discipline-specific research, as an acceptable form of 
publieation for the exeellenee in seholarship eriteria. 

Relations with K-12 and Higher Education Sectors 



Local K-12 Outreach 

CSUN has a strong tradition of engagement with K-12 schools and districts, which many 
respondents cited as a historical consequence of being the dedicated IHE for the San Fernando 
Valley. CSUN is also the training sehool for future teachers in the LAUSD sehools in what is 
now elassified as Local District 2; preservice and in-serviee professional development field 
training offered by CSUN faculty has created strong ties with the district and individual schools. 
Other outreach activities focus on recruitment, such as bridge programs that bring promising 
high sehool students to CSUN to participate in IHE-level scientific research. CSUN faculty 
commonly participate in field days at schools or public events, operate programs to engage K-12 
teachers in laboratory research, and conduct recruiting trips to local high schools and community 
colleges. Generally speaking, according to respondents, these efforts are not intended to recruit 
future math or scienee teaehers, but rather to raise the profile of CSUN in the wider community, 
and to encourage promising students to attend CSUN and go on to further graduate work. 

Eor example, two of the many K-12 outreach programs are Tomorrow’s Scientists and 
the California Seienee Frojeet. Many respondents mentioned Tomorrow’s Scientists. Its goals are 
to reeruit and train future seienee teaehers in the liberal studies Integrated Teaeher Education 
Frogram (ITEF; described in more detail below), and to provide these future teachers an 
opportunity to teach a real science lesson to seventh grade students from a nearby LAUSD 
school. Another program is the California Seienee Frojeet (CSF), a well established K-12 
professional development program funded by the state and the University of California system. 
CSF professional development activities are led by IHE faeulty, and focus on disadvantaged 
schools and districts throughout the state. The CSUN faculty participate in the San Fernando 
Valley Scienee Frojeet, which includes summer institutes in inquiry-based scienee with Saturday 
follow-ups throughout the sehool year. 
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Educational Reform Environment 

At the time of this research, CSUN was in the midst of two major educational reform 
initiatives: a university- wide transformation effort known as the Learning-Centered University 
(LCU) initiative, and the Teachers for a New Era (TNE) project. The ECU initiative is comprised 
of three major elements: (a) maximizing experiential learning opportunities for students, 
especially in the first two years; (b) building a learning community among students; and (c) 
making a sustained and purposeful effort to educate high school students about career tracks. The 
last point, according to a respondent, is “one of the major gaps that haunts the relationship of K- 
12 to universities,” since many high school students have little sense about the variety of possible 
careers and how to select an educational track to acquire the appropriate training. According to 
respondents and documentary evidence, the LCU initiative permeates many aspects of 
institutional practice at CSUN, including announcements for position openings, many CSUN 
official documents and publications, and the orientation session for incoming faculty. Another 
highly visible manifestation of the LCU initiative is the requirement that each department 
develop student learning objectives (SLOs) for each course. 

Teachers for a New Era is funded by the Carnegie Corporation of New York, the 
Annenberg Eoundation, and the Ford Foundation, as a response to critics who charge that IHEs 
are failing to prepare quality teachers (CSUN TNE, 2006). Since teacher training programs are a 
significant aspect of the mission of the CSU system in general, and CSUN in particular, the 
university and the TNE project share a goal to improve the quality of the teacher preparation 
pathways at CSUN. CSUN was awarded a 5-year TNE grant in 2002 that will continue until 
2009 using carry-over funds from the first five years of the award (CSUN, 2006). TNE is a 
comprehensive initiative with many different lines of work, and to adequately account for these 
activities is not within the purview of this case study. TNE and the MSP program have almost 
identical goals regarding improving preservice programs in IHEs by increasing interdisciplinary 
collaboration and improving undergraduate STEM instruction. 

Teacher Education Programs and State Teacher Credentialing Policies 

Of particular relevance to the SCALE and the MSP program are the teacher preparation 
pathways which provide future K-12 math and science teachers their training in disciplinary 
content and pedagogy. The elementary and secondary teacher education programs at CSUN are 
administered by the COE, but STEM and education faculty collaboratively plan and implement 
the preservice curriculum. The major pathways that lead directly to a teaching credential from 
CSUN are described below. Several respondents stated that the traditional credential program (at 
the graduate level) and the liberal studies program (at the undergraduate level) are by far the 
largest teacher preparation pathways at CSUN. 

The traditional credential program is a fifth year post-baccalaureate program in the COE, 
which requires a year of pedagogy-focused coursework and student teaching. Another graduate- 
level option for students at CSUN is the Accelerated Collaborative Teacher (ACT) program. The 
ACT program is a partnership between CSUN and Local District 2 of LAUSD, which provides a 
fast-track credential program for students who are placed immediately at a LAUSD school while 
taking education courses at CSUN. No STEM courses are required for either of these programs. 
One of the undergraduate level pathways is the liberal studies teacher preparation option, which 
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is a program for obtaining a bachelor’s degree only, and is intended for students who plan on 
enrolling in a traditional eredential program or the ITEP junior option (see below). 

Another pathway is ITEP, which is a program for obtaining a bachelor’s degree in liberal 
studies and an M/S eredential eoneurrently. The students are eohorted for this full-time program 
and take at least five eourses in STEM departments. Einally, the four-year integrated program 
(EYI) is a freshman-only program for eoneurrently obtaining a bachelor’s degree and a 
seeondary eredential in the areas of English or mathematies. Of particular relevance to SCAEE 
and the MSP program are the subjeet matter programs, whieh lead to a baehelor’s degree in a 
speeifie diseipline and provide adequate preparation for the state subjeet matter tests that lead to 
a secondary credential (i.e., junior or high sehool). Eaeh subject matter program must be 
specially approved by the state, and at CSETN only the math and geology departments offer this 
option. 



Ill, The Institutional Context 
Significance of Institutional Context 

Beeause reform efforts sueh as SCAEE interaet with various elements of the institution 
and coexist with extant reform initiatives, it is eritical to understand the context in which a 
reform effort unfolds (Patton, 2006; Katzenmeyer & Lawrenz, 2006; Anderson & Helms, 2001). 
Researehers in both K-12 and higher edueation have established that a primary reason 
instructors do not passively reeeive poliey direetives and translate them with eomplete fidelity to 
the elassroom is that they are inelined — if not required — to attend to contextual features of their 
organizations, espeeially the culture of administrative units such as academic departments 
(Birnbaum, 1988; Gamoran et ah, 2003; Coburn, 2001). Insights into the proeesses by whieh 
features of the institutional eontext dynamieally interaet in ways that lead aetors to resist, adapt, 
or adopt reform efforts like SCALE ean improve our understanding of the relationship between 
instruetional policy and local implementation, and provide aetionable knowledge for program 
planners. 



How the Institutional Context Influenced SCALE 

This ease study shows that the faetors that affeeted how the SCALE reform effort played 
out inelude not only degree programs and govemanee struetures, but also material and human 
resources, group identities fostered by struetured interactions, and individual dispositions and 
practices — all of whieh were infiueneed, in turn, by external faetors. Analysis of these faetors 
enables us to understand the often non-linear and sometimes unpredietable interaetions between 
a reform effort, an institution, and its members. With this focus, we emphasize the importance of 
situating an intervention within its local context and of systematically opening up the black box 
of reform implementation by identifying whieh eontextual faetors are aetually salient to the 
reform. We present eontext faetors for SCALE at CSUN using the strueture of the institutional 
context framework. 
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Figure 1. Contextual factors at CSUN relevant to SCALE, 
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This section presents findings on how contextual factors at CSUN interacted with the 
SCALE MSP. Upon analyzing interview transcripts, documents, and field notes from 
observations and interviews, we identified key factors within the institutional context that were 
activated by the SCALE intervention, and played either a supporting or inhibitingrole. We 
assigned a valence to each factor {supporting +, or inhibiting -) based on respondent views or our 
own interpretation of the relationship, and organized these findings by the ICE categories. This 
section is organized by these categories, as is Eigure 1, which summarizes the key findings. 

External Environment 

This category of the ICE includes factors external to CSUN pertaining to institution type, 
national and state education policy, academic training of faculty, economic forces affecting 
education, and local K-12 characteristics. 

Trends in educational reform take root at CSUN (+). CSUN has many grant-funded 
programs aimed at reforming some aspect of institutional life and practice. The primary intent of 
these programs (here called reform initiatives) is to change existing policies, structures, or 
programs. Some of the reform initiatives at CSUN, including the ECU initiative and the TNE 
project, are part of a national trend in educational reform. Several organizations and government 
agencies are supporting educational reform in the STEM disciplines, including the NSE, the 
Carnegie Foundation, and the NRC. This national trend had taken root at CSUN well before the 
SCAEE initiative arrived on the scene, and served to sensitize faculty and administrators to the 
rationale behind reform, develop networks of participants, and generally to lay the groundwork 
for new projects. Of note, some respondents aware of this trend conveyed that the MSP emphasis 
on leveraging the content expertise of STEM faculty to address K-12 educational issues is a 
strategy that is considered insulting and misguided by some education faculty. 

State and CSU system budget crises frustrate faculty yet make external funding 
attractive (+/-). Several respondents observed that the CSU system is currently in the midst of a 
budget crisis, with current or impending cutbacks in hiring, increases in class sizes, and a general 
atmosphere of fiscal restraint. According to one respondent, CSUN is losing positions through 
attrition and reduced replacement rates. Perhaps more relevant to individual faculty and the 
SCAEE goals though, is the increase in average course size, which some respondents cited as a 
major constraint on using active learning pedagogies. In addition, as class size grows, so too do 
the demands of office hours and grading, and some respondents felt that their time to devote to 
research and service activities was reduced by the demands of serving a growing student body. 
One outcome of the budgetary situation is that other IHEs, including regional community 
colleges, may offer better salaries and encourage CSUN faculty to leave their positions. 

However, this situation also serves as a supportive factor for SCAEE, as the pervasive sense of a 
resource-limited environment makes all external funding particularly attractive for administrators 
and faculty. 

Lack of pedagogical training during STEM doctoral work negatively impacts quality of 
teaching (-), Respondents frequently cited the influence of the Research- 1 universities on their 
professional lives. In particular, all faculty respondents noted that during graduate school they 
received no training in pedagogical methods and had to learn on the job. For example, one 
respondent said she had no training in teaching, despite her experience as a teaching assistant. 
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Other than a public speaking course she took in order to do research presentations. An 
administrator cited this feature of research universities, from which most CSUN faculty are 
hired, as having a significant — and unacceptable — impact on the quality of teaching at CSUN. 

STEM teachers are in demand in LAUSD schools (+). LAUSD is one of the largest 
school districts in the country, and hires a significant portion of its teachers from the CSU 
campuses. Los Angeles County currently has a shortage of math, science, and special education 
teachers, and LAUSD is actively seeking teacher applicants in these areas. In fact, through the 
LAUSD Teacher Recruitment and Student Support Grant Program (TRSS), the district is 
offering prospective new hires incentives and reimbursements of up to $18,000 to teach in low- 
performing schools in these subject areas. A respondent noted that student teachers get snapped 
right up and that local districts want more graduates, which is problematic because CSUN 
enrollment figures in the COE are dropping. Teacher shortages in LAUSD increase the value to 
both K-12 and IHE training institutions reforms focused on STEM preservice teachers. 

Comprehensive universities are closely aligned with MSP goals (+). The missions and 
programs of comprehensive universities such as CSUN are closely aligned with the goals of the 
MSP program, including a focus on teacher training, undergraduate education, and developing 
IHE/K-12 partnerships. Eor example, teachers at EAUSD participate in several CSUN- 
sponsored professional development programs, and many COE faculty have close ties with 
administrators and teachers throughout the system. The district and CSUN are further connected 
since most of the students at CSUN are from Los Angeles County and are graduates of LAUSD 
schools. Several respondents also noted how CSUN’s close affiliation with the K-12 sector 
serves to engender a keen awareness of the interrelatedness of the local K-12 sector and CSUN: 
many CSUN students come from the schools and communities surrounding the university, and 
return to live and work in those communities. Thus, in contrast to many research universities 
(e.g., UW-Madison), CSUN personnel were already conversant with the goals of the MSP 
program and the university had an established track record in these areas. 

CA teacher credentialing policies discourage interdisciplinary collaboration (-). CSUN 
teacher preparation programs are built around the state’s teacher credentialing system, which 
requires a broad liberal arts education for elementary teachers or an undergraduate degree in a 
discipline for secondary teachers, coursework on pedagogy, and student teaching. However, the 
state policies governing teacher training create a division of labor between the COE and College 
of Natural and Behavioral Sciences (CNBS) regarding teacher preparation, such that pedagogy 
and content are addressed in separate degree programs. The Division of Teacher Education is 
housed and administered by the COE, and includes the undergraduate liberal studies program, a 
post-baccalaureate teacher education program, and a graduate program, while content-based 
coursework are housed and administered by the CNBS and their respective departments. In most 
cases, there is little coordination between the two colleges in order to provide students a coherent 
and structured preservice experience. 

One of the odd and frustrating things at this university is the gigantic gap between the 
undergrad [coursework and graduate] credential programs. Despite all this work together 
[on TNE], it may actually not do a thing for our teacher candidates, [because] as 
undergrads all they’re doing is majoring in their subject. So they’re a math major. And 
they can opt for this math education strand but it doesn’t mean anything credential wise. 
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It just means that they’re taking a set of courses that’s more oriented towards education 
than others, but when they walk out of the door they’ve got a math BA or BS and nothing 
else. Now, they can then choose to enter a credential program, which if they come to 
CSUN is in the College of Education. And they only get one math-specific course at that 
point, and that’s the math methods course. It’s unusual for an undergrad to come out of a 
CSUN math major and go right into the credential program. We don’t see that continuity 
at all. It’s more likely that they’ve come from the workforce where they’ve been for the 
last 10-15 years. It’s a pretty small pipeline immediately back to us. (COE faculty) 

It must be noted that CSUN has several programs and initiatives that address this 
predicament, including the subject matter programs in math and geology and the ITEP programs 
(see below). However, the prevailing condition is that of a bifurcated preservice system that is 
more aligned with the state requirements and less aligned with the MSP goals of a coherent 
preservice pathway for future STEM teachers. 

Policies governing the type and quality of in-service PD for K-12 teachers are lacking 
(-). Although California has content standards and a curriculum framework for K-12 schools, it 
has few policies or regulations governing the type and quality of PD offered to in-service 
teachers. Instead, districts, based on their own criteria, ultimately decide how best to provide 
instructional support and PD. Since these criteria usually include compliance with state content 
and curricular standards, most PD workshops and seminars used by a district closely align their 
content with these standards. The only two programs that have some policy leverage over PD are 
the California Beginning Teacher Support and Assessment (BTSA) and the National Board for 
Professional Teaching Standards, both of which include professional development requirements 
for the completion of their respective licenses, and have general guidelines for PD quality. In the 
case of the BTSA, K-12 districts have the authority to select these approved vendors. Thus, even 
if SCALE and other MSP programs provided high-quality PD, there is no way to ensure that 
LAUSD teachers would not participate in other PD opportunities, some of which may not be 
aligned with the principles of active learning promoted by SCALE. 

Dynamics between IHEs and K-12 educators can be tense (-). Some respondents 
indicated that in many cases IHE and K-12 educators suffer from problematic dynamics, based 
largely on the sentiment that people in the ivory tower have little understanding of what life is 
like in the trenches. Thus, when faculty make suggestions about how to teach or manage their 
classrooms, some K-12 educators bristle and resent being told what to do. Some STEM and 
education faculty noted, however, that this is an over-generalization and that many faculty have 
taught at the K-12 level and are committed to improving that sector through their research and 
preservice training. Another by-product of this dynamic is that faculty, particularly STEM 
faculty, who exhibit empathy and respect for their K-12 colleagues are highly valued and in 
some cases considered unprecedented. This situation is particularly salient to SCALE efforts that 
bring IHE and K-12 educators together, as this is a widely agreed upon backdrop to such 
interins titutional efforts. 

Internal Structure 

This category of the ICE includes factors pertaining to the internal structure of CSUN, 
including the organizational structure (governance, teacher education programs, STEM degree 
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programs), student body composition, instructional workforce composition, personnel policies, 
leadership, and active and historic reform initiatives. 

Governance system and autonomy exert strong influence on faculty (+/-). The 
governance system of CSUN, and its emphasis on departmental and faculty autonomy, was cited 
as a factor that strongly influences faculty practices at CSUN. This system favors departmental 
and faculty decision-making control regarding academic programs and related policies. Faculty 
autonomy can serve as a supportive influence to SCALE goals by allowing faculty significant 
leeway in how they prioritize their professional lives and make instructional decisions. However, 
this autonomy may also work against SCALE goals as faculty are under no obligations to adopt 
inquiry-based teaching practices or collaborate with their colleagues in the COE. 

CSUN mission supports undergraduate education and teacher preparation (+). As the 
former San Eemando Valley College, there is a strong history of connection with the local 
community, which has made its way into the institutional saga. Several respondents explained 
that CSUN has a historic and primary identity as a community-based institution, and that this 
history and identity give the university pervasive influence in the community. A respondent who 
recently relocated from a research university found a strong culture of teaching and was a bit 
blindsided by the intensity and coherence of the faculty and administration’s commitment to its 
students and teaching. Many faculty also expressed the sentiment that the institution has a 
historic and moral obligation to serve the needs of K-12 schools in the area. Despite this feeling, 
we heard conflicting perspectives about the viability of realizing this obligation, due to workload 
pressures, and complications arising from working with as complex and politicized a district as 
LAUSD. 

Multi-tiered leadership generally favors pedagogical reform (+/-). As at other IHEs, 
several layers of leadership influence the day-to-day operations of CSUN, each with its own 
unique leadership positions. These layers and positions include the upper administration (e.g., 
provost’s office), college level (e.g., dean’s and associate dean’s), and departmental level (e.g., 
department chair and particularly influential faculty). CSUN provides varying degrees of support 
for pedagogical reform at these different levels, which may result in somewhat inconsistent 
messages delivered to faculty. According to respondents, these mixed messages have included 
some decisions about faculty tenure and promotion that vary in their evaluation of the value of 
pedagogy-based activities, and divergent public declarations about the relative value of research 
and teaching in faculty life. As a result, the supportive leaders provide a supportive element to 
projects like SCALE, but the varied support at different administrative levels may contribute to a 
confusing and potentially discouraging institutional environment for faculty. 

Pipeline of preservice majors for CSUN science departments is limited (-). According to 
some respondents, while the lack of science majors at CSUN severely restricts the pipeline of 
future high school science teachers, more alarming is the low number of existing science majors 
who are considering teaching K-12 as a career option. One respondent observed that while math 
has a large cohort of students who plan to teach high school math, there is not a similar 
population for science. In the sciences, students generally plan on becoming doctors or entering 
industry in some fashion. This problem is noted in a TNE report, as follows. 
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It must be strongly stated at the outset that single-subject science faces a unique challenge 
in the preparation of teachers in secondary education compared to those in mathematics, 
the arts and humanities, and social sciences. The population of baccalaureate candidates 
whose stated career goal is to become a high-school teacher of science is very small in 
the departments of Biology, Chemistry and Biochemistry, Geological Sciences, and 
Physics and Astronomy. Because of the small numbers, it is currently not feasible to 
create a curriculum specifically for prospective teachers of high-school science. The 
problem is not unique to CSUN. The severe shortage of high-school teachers of science is 
a highly publicized national problem. The number of students majoring in science is 
dwindling, and most of these majors are planning careers as professionals in industry or 
higher education. (CSU Northridge, 2006) 

Other respondents corroborated the sentiment that students generally “don’t come to 
choose a science major to be a teacher.” Several respondents stated that low enrollment in 
chemistry, physics, and geology was a major concern for reasons of funding and the long-term 
health of the departments. This presents a considerable problem for projects such as SCALE that 
hope to influence a cohort of preservice students: this cohort may not actually exist. In response, 
many faculty and administrators in the CSUN STEM departments are focused on recruiting 
students from area high schools and community colleges. Respondents noted that these 
recruitment efforts are going beyond traditional science fairs and other K-12 outreach to a more 
targeted focus on aligning the pipeline of students from community colleges and high schools to 
CSUN. Eor example, as one respondent noted, CSUN is exploring a dual admission system with 
community colleges where students take the correct introductory courses while working directly 
towards a CSUN degree. 

Math and geology subject matter programs require inter-departmental collaboration 
(+). One of the requirements for obtaining a single-subject (secondary) teaching credential in 
California is to provide evidence of subject matter proficiency, which can be done by completing 
an approved IHE subject matter sequence, or by passing the California Subject Examinations for 
Teachers (CSET). As noted above, only two departments at CSUN have subject matter 
programs, math and geology. These four-year programs build in all of the coursework required to 
pass the CSET. One respondent stated that without this type of preparation, a student with a math 
or science baccalaureate would need to take another semester of coursework to get adequate 
breadth in their subject area. Another respondent noted that CSUN used to have more subject 
matter programs in the sciences, but found that few of the students enrolled in the program 
planned to get their bachelor degrees at CSUN or to obtain a teaching credential. Instead, they 
were using CSUN as a “fly-by” in order to avoid taking the CSET, and thus consumed 
considerable administrative resources in the process. These programs entail close collaborations 
between STEM and education faculty in creating course curricula, and pay close attention to 
fusing content and pedagogy. As a result, these programs represent probably the best example of 
a blended program leading to a secondary science or math teaching career, and some respondents 
hope that the eventual success of these programs will be a major recruiting point in the future. 

Large part-time and non-tenured workforce complicates efforts to reform STEM 
instruction (+/-). Several respondents noted that the growing trend in U.S. higher education to 
hire non-tenure track faculty, especially to teach lower division courses, has important 
implications for reform efforts such as SCALE, since the MSP focus is on full-time faculty. 
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However, some respondents noted that many leeturers and adjunet faeulty have been at CSUN 
for several years, and are deeply integrated into departmental operations. In fall 2007, 61% of the 
faeulty positions at CSUN were held by part-time instruetors, and only 39% by full time 
instructors (CSUN, 2008). The part-time faculty are not on the tenure track and play significantly 
different and less influential roles in departmental policy-making and administration than full- 
time faculty. Thus, for important leverage points that may influence STEM instruction, such as 
syllabus development and student assessment, part-time faculty may have less influence than 
their colleagues. Yet, because these faculty are not subject to the pressure of obtaining tenure, 
they may be more likely to be able to participate in a reform initiative. 

Departmental tenure and promotion policies discourage teaching innovation (-). 

Several respondents, including faculty and administrators, noted that across the CSU system 
faculty are experiencing increased pressure for research and publication productivity. As one 
respondent noted, the goal is to bring the standards of faculty scholarship more in line with 
research institutions. Faculty respondents generally felt that this shift in policy is placing further 
demands on their limited time, and reducing their prospects for participating in service activities 
such as SCALE. As one respondent noted, “You don’t get published for participating in a grant,” 
and since many faculty use summer breaks as an opportunity to conduct research and write 
journal articles, summer professional development for K-12 teachers was cited as a relatively 
low priority. Furthermore, for faculty who are actively engaged in pedagogical research, 
respondents stated that administrators generally do not understand that educational research takes 
longer than traditional scientific research, which means that faculty who participate in this type 
of research will be less productive than others in terms of articles submitted and published. 

Given these pressures on faculty, several respondents noted the need for two types of support: (a) 
release time afforded primarily through grants and also through committee work; and (b) support 
and mentorship in writing articles and conducting research. 

Workload is heavy and not amenable to service (-). Several respondents, including both 
faculty and administrators, observed that the teaching load in the CSU system is extremely high 
compared to many IHEs, making research and service activities difficult to accommodate. Given 
the prominent role that research and publications play in tenure and promotion considerations, 
faculty usually allocate any time beyond their teaching responsibilities to research and/or 
writing. One respondent who recently relocated to CSUN from another university called the mix 
of teaching load and research expectations at CSU a “worst nightmare.” Several respondents 
described their workload as overwhelming, and used the words like “frazzled” and “a struggle” 
to describe their professional lives. A common element across all of the respondents was a 
perceived lack of time to accomplish all of the tasks that they wished to. Some faculty also 
expressed some anger towards the CSU and CSUN administrations. Faculty explained that with 
class sizes growing and with increasing demands to publish scholarly works, they also were 
expected to participate in the ECU initiative and to focus on assessment issues. As one 
respondent stated, “You can’t get blood out of a turnip.” Another criticism expressed by some 
respondents was that, while administrators stated that they would treat pedagogy-based research 
as scholarly work, they did not understand (as noted above) that education research generally 
takes longer, and is less tidy than traditional scientific research. Thus, a TNE faculty member or 
others who are conducting pedagogy research may not be as productive as other faculty when it 
came to publications. 
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According to several respondents, faculty are usually only able to aceommodate service 
activities such K-12 professional development workshops into their heavy workload when they 
receive release, or are assigned time from department or college administrators. Generally, 
assigned time is granted for administrative duties, sueh as serving as department chair, or 
through grant and/or university funding that buys out a portion of the faculty member’s teaehing 
load. This enables faculty to participate in activities such as TNE or SCALE without requiring an 
unsustainable level of effort. In cases where assigned or release time is not available, faeulty 
generally partieipate in serviee activities only during the summer break, which is also an ideal 
time for research activities. 

Math department strongly divided on pedagogical reform (+/-). CSUN’s math 
department has long been split by the controversy over constructivist pedagogical strategies in 
mathematies, also known as the math wars. Numerous respondents noted that the atmosphere in 
the mathematies department is tense due to the politieal and ideological nature of the debate, 
which some felt had long ago ceased being a collegial discussion over methodology. One 
respondent observed that the division, which is not unique to CSUN, is sad because all of the 
faeulty are committed to math education and improving student learning. The division manifests 
itself in two ways relevant to SCALE: the revision of the liberal studies math eurriculum, and 
relations between IHE faculty and K-12 personnel. Many respondents were heartened by recent 
collaborations between the COE and the small cohort of pedagogy-minded math faculty who are 
revising the liberal studies curriculum to infuse pedagogieal content knowledge into the 
eurrieulum. Some respondents indicated that reform efforts such as SCALE should only engage 
the faculty in this cohort. 

Unfortunately, several respondents reported that the efforts of this cohort are being 
aetively ehallenged by another cohort of math faculty who disagree with the eonstructivist 
approaeh. One faeulty member stated that the best argument for reforming an entire STEM 
department is to avoid situations where a group of faculty actively undermines the efforts of 
others. In addition, some respondents noted that the debate has resulted in real harm to IHE and 
K-12 relations, since it has left K-12 teaehers feeling defensive about their own teaehing styles, 
and confused about the best pedagogical approach to employ in their classrooms. 

Existing pedagogical reforms set the stage with support and skepticism (+/-). As 
previously noted, several nation-wide pedagogieal reform efforts are addressing STEM 
edueation and preserviee training, and that three sueh efforts — TNE, CSP, and ECU — already 
were underway at CSUN. These reforms set the stage for SCALE by familiarizing faculty with 
the core issues (e.g., active learning strategies, interdisciplinary collaboration), creating networks 
of STEM and education faculty, and initiating programs and activities that were aligned with the 
goals of SCALE. For example, one of the aetivities of the TNE project was an attempt to hire 
tenure-track faculty who would focus on STEM pedagogy and outreach in their departments. By 
doing so, the TNE project would guarantee itself a long-term presence and influence at CSUN, 
particularly since most of the hires had unique Memoranda of Understanding that officially 
sanction their focus on pedagogical research and K-12 activities. 

However, the flip side of such precedent is that some faculty were skeptical of the 
rationale behind such reforms, and were less inclined to think highly of SCALE. For example, 
the TNE projeet had some difficulties in aetually hiring these pedagogy-foeused faculty, as some 
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departments disagreed on the responsibilities of the position or even the basie premise of the 
TNE project. Furthermore, some respondents noted that being identified as “the K-12 expert” 
has negative consequences. 

Already some people in the department weren’t particularly excited about TNE and don’t 
really feel as if it’s their department’s business to be worrying about making teachers. 
Then, they’re unwilling to give up, let’s say, a position to bring someone in whose 
research work is going to be education. Now interestingly, I don’t thi nk it’s giving up a 
position, I think that position was extra. (STEM faculty) 

Reactions to ECU among faculty respondents were generally supportive regarding the 
intent of the initiative, but mixed regarding its implementation and ultimate impact on faculty 
life. Some reported that they and their colleagues are still not sure precisely what it means, and 
others view it as an additional burden from the administration. Considering the overwhelming 
workloads, one respondent stated that some resistance is not surprising, and that change doesn’t 
happen in higher education without resistance. This respondent noted that “at first blush, the 
ECU shifts responsibility to the student, but it requires significant work by faculty.” Despite 
these challenges, most respondents report that there is significant buy in from the faculty for the 
ultimate goals of the initiative. They explained that this buy-in is concomitant with the 
knowledge that current CSEIN students may soon be teaching future CSETN students, and thus 
faculty had a vested interest in ensuring that they were being taught well. 

Resources 

This category of the ICE includes factors pertaining to material resources (e.g., time, 
funding), and social resources (e.g., networks). 

Active and historic reform efforts fostered social network of STEM educators (+). As 
previously noted, the STEM disciplines have a strong history of participating in educational 
reforms, and these active and historic efforts have fostered networks of STEM educators and laid 
the groundwork for future projects such as SCAEE. The funding and employment opportunities 
afforded by these projects played a significant role in developing the readily identifiable cohort 
of faculty who are particularly active and experienced in STEM education. This community is 
recognized by K-12 personnel as a valuable resource when they need collaborators or expertise 
pertaining to STEM education. In a similar fashion, personnel at CSETN recognize that there are 
people on campus who have experience in working with pedagogical issues and in the K-12 
sector. It is worth noting that these networks in many cases extend beyond the confines of 
CSUN, as many personnel have extensive personal networks within their disciplines and 
specialty areas. These networks provide a resource for sharing ideas and building knowledge, 
particularly through professional meetings and electronic communications. 

One nice thing about CSUN is being in this Teachers for a New Era program, [where we 
have] 30 or 40 people [in science and math] who are dedicated to improving teacher 
education from all different perspectives and it’s wonderful. It’s a great place to be 
because there’s nobody slamming the door in my face saying, “I can’t be bothered.” 
There are always people that want me involved because they have another idea and so, to 
me, it’s a great place to be doing this right now. (STEM faculty) 
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Some respondents singled out the example of Dr. Steven Oppenheimer, who is a eaneer 
researeher in the biology department at CSUN, and a leading proponent of engaging K-12 
teaehers and students in STEM researeh. Aecording to these respondents, Dr. Oppenheimer has 
established positive relationships with several K-12 districts, which gives other faculty 
immediate credibility with the K-12 personnel. 

K-12 professional development services already exist at CSUN (+). Members of the 
previously mentioned cohort of faculty devoted to STEM education often design and facilitate 
professional development programs for K-12 teachers. In addition, faculty in the COE regularly 
provide professional development for K-12 teachers through a variety of programs and thus 
serve as a resource for K-12 districts. These entities, their skilled staff, and their positive 
reputation in local K-12 districts, each serve as a supporting factor to the SCAEE project. 

Funding pressures limit resources (-). As previously noted, the state and CSU system 
budget crises are deeply affecting faculty at CSUN. As a result, there is a consistent financial 
pressure on all CSUN personnel to compensate for declining state funds by obtaining external 
funding to sustain or even enhance current teaching programs and research capacity. CSUN lacks 
additional staff and other resources available to adequately instruct undergraduate students. 

Under pressure to increase enrollment, departments often respond by just “churning out 
undergraduate students,” so that faculty can devote more time to seeking external funds to build 
research programs, developing a track record for fundraising, and accomplishing other tasks for 
which college and departmental funds are not available. 

Student body requires more remedial courses. (-). Several respondents raised the issue 
of inadequate student preparation as a major factor influencing their work at CSUN. While some 
respondents viewed the generally poor level of preparation of incoming students as an 
unavoidable aspect of living in a large, urban area, they noted that CSUN has had to address the 
“realities of student abilities,” by increasing the amount of remedial coursework for incoming 
students. One administrator noted that students who come to CSUN with no more than 10th 
grade math (Algebra 2, a California state requirement), or after having been out of school for 
several years, “generally flounder.” This is supported by the fact that over 80% of incoming 
students fail the introductory math assessment. One respondent noted that the mathematics 
preparation of incoming freshmen was “sad,” and that students lack “sub-high school skills” such 
as adding fractions or basic geometry. Some respondents argued that the increase in remedial 
courses drains departmental resources and further increases the teaching challenges they face. 

Status and social capital influences ability to participate in reform (+/-). Some 
respondents noted the importance of their tenure status and associated social capital (e.g., social 
status, authority, and reputation) in enabling them to participate in reform activities. These 
respondents stated that junior faculty must focus on teaching first and then research and 
publications in order to adequately develop a portfolio for tenure and promotion. Thus, the RTF 
policies of the department and their inherent authority play a powerful role in shaping how junior 
faculty budget their time. 

And it’s easier for me to do all this because I’m such a senior faculty. Whereas a lot of 

junior faculty that get involved, have been involved in SCALE, there’s actually more 

pressure on them because of all these different requirements, but also there’s more 
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pressure in general for new seienee faculty at CSU. More publications, more grants. 

(STEM faculty) 

Furthermore, respondents noted that if a faculty member’s disciplinary reputation is 
sufficiently high (as measured by high-prestige research, grant funding, and publications), then 
their colleagues may be more forgiving if they become involved in K-12 outreach or 
pedagogical reform. 

Pervasive Values 

It is important to note that we do not claim that all faculty, administrators, and staff at 
CSUN exhibit the following pervasive values. However, because the values presented here were 
expressed by many respondents from different social groups and administrative units, it is 
reasonable to surmise that these values are widely shared by many people at CSUN. Moreover, 
each individual may have personal interpretations or versions, shaped by backgrounds, 
personality, social position, discipline, and other situational or demographic variables. 

CSUN’s teaching mission and disciplinary traditions are in tension (+/-). One of the 
defining features of CSUN is the oft-cited tension between the teaching mission of the university 
on the one hand, and the disciplinary traditions and their attendant norms and practices on the 
other. 



I would say [undergraduate education] is a priority, but the big priority of the department 
and the college is research. I mean, even though we’re an undergraduate teaching 
institution, and our primary goal is to teach — because we’re assigned twelve units of 
teaching every semester, research is just being piled on top of that. (STEM faculty) 

As the former San Fernando Valley College, there is a strong history of connection with 
the local community, which has made its way into the institutional saga. Several respondents 
explained that CSUN has a historic and primary identity as a community-based institution, and 
that this history and identity give the university pervasive influence in the community. Many 
faculty also expressed the sentiment that the institution has a historic and moral obligation to 
serve the needs of K-12 schools in the area. Despite this feeling, respondents expressed 
conflicting perspectives about the viability of realizing this obligation given the demanding 
workload (see below) and the disciplinary traditions that often worked against this teaching and 
service oriented mission. 

[At CSUN faculty] will be pulled two ways. [First], is the department, which is locally 
situated in a region that has a local habitation and a name and demography to it. They’ll 
be pulled that way by asking about their classes. [Second], they’ll be pulled abstractly on 
the other axis towards the discipline. This is a hierarchy of performance and achievement 
that is national and international in scope and standard, and the farther it removes itself 
from contamination with the region, the more successful people are, just given the way 
the hierarchy works. So what we can provide here is a route to success that we reward if 
we try to bridge those two things. But I think if you’re interested in the career 
ladder. . .you have to subscribe to the axis of disciplinary research. So I think that’s a real 
tension that people feel as they move up the line. There’s some people who can 
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overpower it and deal with that, but I think, that’s where a lot of people get oppressed and 

concerned. (Administrator) 

The idea that the disciplinary axis influences faculty regardless of their geographic and 
institutional situation was often credited to doctoral programs, which both train students to 
become technical experts in a field and socialize them into discipline-based behavioral and 
intellectual norms. Further reinforcing these norms and values is the fact that once students are 
awarded the Ph.D. and granted entry to a discipline, their discipline becomes the primary source 
of their identity as a professional academic. While the predominance of the disciplinary value of 
research accomplishments serves to inhibit the goals of SCALE, the teaching oriented mission of 
CSUN serves to support SCALE. The resulting tension has important implications for 
interdepartmental collaborations and for dictating how individual faculty members prioritize 
their workload, both of which are critically import to the SCAEE project. 

Scientific legitimacy and credibility equated with basic research accomplishments (-). 
Perhaps the most frequently cited aspect of faculty life at CSUN was the demanding workload 
and its implications for the prioritization of research, teaching, and service activities. This 
conception of the priorities is a function of both CSUN’s RTP policies and the way disciplinary 
expectations and traditions are instantiated at CSUN. No STEM respondents had prior training in 
pedagogy in their graduate training. Instead they received training that consistently placed 
higher value on conducting research and acquiring expertise in a specialty area of their field. The 
lack of training in pedagogical issues often results in a reliance on teaching “the way I was 
taught,” which was described as a practice that “was taken for granted.” A practical corollary of 
this factor is the departmental practice of separating teaching and research activities. Eor 
example, some STEM departments have hired faculty members specifically to focus on outreach 
and education — in effect, to not pursue research in the discipline. However, some respondents 
emphasized that even though colleagues may not be particularly interested in STEM education, 
they find it useful to have somebody around who is proficient in these matters; such people help 
the research oriented faculty address the broader impacts criteria required by NSF and other 
funders, and enable the other faculty to focus on their research. 

Both STEM and COE faculty feel pressure to conduct research. The importance of 
achieving legitimacy and credibility in a particular discipline is viewed as a critical ingredient to 
academic success in general, and for interdepartmental collaborations in particular. Competence 
in a discipline and specialty area is made legitimate and credible by a faculty position, 
publications, and an active research program. This said, greater credibility is often ascribed to 
faculty in STEM disciplines than in education. This is evident from statements indicating that 
STEM faculty and academic staff who are engaged in educational activities need impeccable 
STEM credentials in order to the shore up their legitimacy with disciplinary colleagues. The high 
value placed on STEM disciplinary credibility also is evident in the fact that STEM faculty often 
question the STEM discipline credentials of faculty who specialize in math and science 
education in the School of Education. Respondents also indicate that academic staff, even those 
with impressive academic background and achievements, do not fit into the established hierarchy 
of higher education achievement and thus may be less desirable than faculty for projects such as 
SCAEE. 
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Distinction between hard and soft sciences fuels long-standing tension between 
departments in the COE and some STEM departments (-). A salient corollary of faculty 
members’ close professional identities with their disciplinary traditions is that different faculty 
may hold value systems that differentiate disciplines in a hierarchical order. In particular, 
academics commonly distinguish between the hard and soft scientists, where the hard sciences 
are considered more rigorous, as they take an approach based on reproducible experimental data, 
and the soft sciences are considered more interpretive, and anecdotal. This distinction is evident 
in the following comment: 

A gap still persists between education and the social sciences on the one hand and math 
and science on the other. [STEM faculty] all understand that observation is mediated by 
instruments and no one’s going to argue that piece at all with you. But there’s still this 
belief that science fact and science theory is out there independent of perception, their 
understanding of pedagogy is that it is just there and can be apprehended and should be. 
And you could explain to them that that contradicts their other attitudes and they’ll say, 
“Yes, it does,” but it doesn’t really matter, they’ll just go back to it in some ways. So I 
think that’s why you have to sort of go back to, deal with the training issues that are 
involved with that. (Administrator) 

Attempts to engage STEM and education faculty in joint efforts to improve the 
curriculum and pedagogy for preservice teachers are hampered by a historical and persistent 
mistrust between these groups. This division is evident in the way one respondent explained that 
TNE faculty in STEM departments use the TNE initiative as a “way to give cover to someone in 
Arts and Sciences to work on education.” Most respondents in the COE observed that throughout 
their academic careers they have regularly been treated as sub-par, mostly because education is 
viewed as a “soft” and “applied” science. On the other hand, several respondents stated that the 
division between the two colleges was not as bad at CSUN as at their previous institutions, 
where STEM faculty rarely interacted with education faculty. The existence of several 
intercollege collaborations, particularly for the liberal studies program, are testimony to the 
existence of a cohort of faculty from both colleges who have overcome this division. 

Faculty disagree about the importance of content and pedagogy in preservice STEM 
courses (+/-). Respondents made clear that within certain departments there are major 
disagreements about the courses required to effectively train a K-12 teacher, and the approach 
that the university should take in designing and offering these courses. This issue is particularly 
touchy in math. For example, some STEM faculty have responded to the problem of declining 
mathematical performance by advocating an increase in the number of content courses required 
for preservice teachers, with a subsequent decrease in the number of methods courses. 
Furthermore, the selection of the appropriate content for preservice teachers is not a simple 
matter. Some respondents observed that a reliance on the core content courses as currently taught 
in STEM departments is a mistake, since research indicates that accumulating courses in a major 
does not automatically make someone a good teacher, and that the problem is that there is no 
“connection between the content preparation and the pedagogy.” Because agreement on these 
matters is required before effective action can be taken, we assess this disagreement as exerting a 
negative effect on SCALE efforts. 
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Individual Sense-Making 

This category of the ICF includes various IHE -related elements of an individual’s sense- 
making process, including workload considerations, status, background and training, and 
personality. 

Funding pressures and workload exert strong pressures on faculty time (+/-). As 
previously noted, demanding faculty and academic staff workloads present a major barrier to 
participation in a STEM education project such as SCAEE. Eaculty responsibilities also include 
research and publishing, teaching, advising, service, and miseellaneous duties ineluding writing 
proposals. Thus most faculty and staff are unavailable to participate in STEM edueation reform 
activities, unless a person has a grant that can buy out teaching responsibilities, holds a position 
that requires participation in STEM education, or is extremely motivated. Most faculty 
respondents stated that while they struggle with their various professional responsibilities, 
invariably their top priority was ensuring that they were adequately prepared to teach. They 
conveyed that the primary factor motivating this decision is a personal commitment to high 
quality teaching. They also noted other factors such as explicit encouragement from the 
administration and the implicit pressure to adhere to the institutional eulture of teaching. 
However, this desire is in eonstant tension with faculty’s other responsibilities, partieularly those 
pertaining to research and its implications for an individual’s tenure and promotion prospects 
(see above). Thus workload demands are a eritieal interpretive frame used by CSUN personnel 
when assessing participation in STEM education activities. This said, faeulty emphasized the 
importance of grants such as TNE (which reduces the teaching load), or SCAEE (which provides 
stipends) as a factor that makes service activities possible for all faculty, regardless of seniority 
status. 



Faculty status influences time management (+/-). Decisions about time management are 
a particularly important issue for junior faculty who are under pressure to “publish or perish.” 
Respondents noted that junior faculty must focus on teaching first (in order to get through the 
day adequately), and then on researeh and publications. Once tenure is achieved, respondents 
observed that the balanee of responsibilities may shift again, as faeulty either focus entirely on 
their research to the detriment of their teaching, or become passionately engaged in teaching or 
service. The common theme here is that senior faculty with tenure have much more freedom to 
decide how they allocate their time and resources. It is also important to note that faculty who are 
not on the tenure-track generally experience other pressures. While they may remain extremely 
busy with teaching and research responsibilities, the pressures of tenure do not influence their 
decisions. 

Various personal reasons motivate engagement in reform efforts (+). Respondents 
identified a variety of personal, politieal, and praetieal reasons for becoming involved in K-12 
activities. Some faculty from both STEM and education field were former high school teachers 
and were predisposed to involvement in pre- and in-service activities. In addition, some of these 
faeulty were direetly engaged in K-12 edueation through their researeh or their teaching 
responsibilities in the seeondary or elementary edueation programs. Several STEM faeulty noted 
that they had a particularly innovative professor in their graduate training, which led to an 
intellectual curiosity about teaching and learning. In any case, respondents indicated that they 
became engaged in programs like SCAEE for diverse personal reasons. 
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Intellectual curiosity and willingness to take risks (+). Willingness to learn and go 
outside of one’s comfort zone emerged in interviews as an important element of personal 
disposition that affected willingness to engage in a K-20 education improvement activity. 

Among the reasons expressed for participating in SCALE and similar projects was personal 
desire to learn about a new topic (whether the K-12 curriculum or STEM pedagogical content 
knowledge), and to change departmental policies governing teacher education. 

Practices 

This category of the ICE includes factors pertaining to an individual’s classroom 
instruction (e.g., planning and delivery) and task-based collaborative activities with both IHE 
and K-12 partners. 

Some STEM faculty maintain a didactic approach to STEM instruction (-). Some 
respondents expressed the view that their approach to pedagogy is informed and influenced by 
their own pedagogical training, or lack thereof, and in many cases replicates the approach of 
their mentors. Many respondents observed that the traditional teaching model used in IHEs is the 
transmission of content, in which the instructor conveys a body of information, usually through a 
lecture, with the expectation that the students will absorb the information. Some respondents also 
noted the difficulty in adopting a more student-centered approach, given the need to cover a 
prescribed set of content in a given class. In light of research on learning, this factor has a 
negative effect on projects such as SCALE. 

Some STEM faculty experiment with an inquiry-based approach (+). An extension of 
the presence of a STEM education cohort at CSUN is that some STEM faculty were conversant 
with or at least are sympathetic to pedagogical reform in math and science. Although these 
faculty vary in their relative levels of expertise with specific theories and methods for inquiry- 
based instruction, their reported teaching practices are a factor that is supportive of projects like 
SCAEE. However, even for these faculty there is a stated need to balance the desire to employ 
techniques such as group work with the need to cover a certain amount of content in a short 
period of time. 

Respondents report participation in interdisciplinary activities (+). Several respondents 
cited a cordial and active collaborative relationship between STEM and education faculty, 
particularly for liberal studies which has several active cross-college faculty committees that 
successfully design curriculum and develop programs. Eor example, an earth sciences faculty 
member teaches a biology course for the ITEP freshman cohort, and coordinates with an 
education faculty member who teaches a concurrent science education course that is required for 
the students. Together, the two courses are intended to fuse the disciplinary content with 
appropriate pedagogical methods. 

However, respondents indicated that outside of the liberal studies program, STEM faculty 
have limited opportunities to participate substantively in preservice programs. This is especially 
the case for the single subject credential, where there is little to no collaboration between the two 
colleges. According to several respondents, this lack of collaboration is due not to overt hostility 
between the colleges, but rather to lack of a rationale to collaborate. Two exceptions are in the 
math department: the EYI math program, and the math education option for undergraduate math 
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majors. For these programs, a eohort of math faeulty works with math edueators in the COE to 
revise eourses, infusing pedagogy eontent knowledge into eourses. For example, the university 
eommittee on edueator preparation and the liberal studies eommittee both inelude members from 
departments across campus. Some respondents cited these committees as providing a unique 
vantage point on university happenings that is not available to other faculty. 

Respondents report active participation in K-12/IHE collaborations (+). Respondents 
reported numerous collaborative activities between CSUN and local school districts, and noted 
that these support SCALE efforts. These activities include both structured and unstructured 
collaborations. Unstructured collaborations include informal faculty interactions with LAUSD 
schools, usually in the form of math or science outreach efforts. As described above, there are 
several examples of structured and long-term programs at CSUN that involve K-12 educators 
and/or students in one way or another. These programs, especially the teacher preparation 
programs in the COE, and professional development programs such as the CSP, are key elements 
of CSUN’s reputation as an IHE that is intimately involved with the K-12 community. 

According to respondents, awareness of and identification with this reputation is widely shared 
by the faculty, and STEM faculty, who themselves are not engaged in K-12 activities but who 
are acutely aware of their institution’s reputation and mission. We assess this factor as having a 
positive affect on efforts such as SCALE. 

IV, Findings on the SCALE Intervention 

This section presents a summative evaluation of SCALE activities at CSUN, consisting 
of descriptions of the activities from March 2005 to August 2007, observed outcomes of these 
activities, and analyses of the longer-term consequences of each intervention. We summarize the 
project’s background, goals, activities, and outcomes and discuss the influence of the 
institutional context on SCALE. This discussion more realistically links project outcomes with 
contextual factors, thereby avoiding the problem of presenting outcomes as if they function in an 
institutional vacuum, independent of their context (Anderson & Helms, 2001). 

At the time of the SCALE intervention, there were at least three other educational 
reforms efforts underway : the provost’s ECU initiative, the TNE initiative, and the CSP. Since 
many SCALE participants were also involved in these other projects, it is difficult, if not 
impossible, to isolate the specific effects of SCALE on these individuals. Similarly, since each 
project was exerting pressure on various aspects of the institutional context of CSUN, it is 
challenging to identity where the effects of SCALE begin and of other programs end. As a result, 
some of the findings described in this section may be the result of any combination of these 
initiatives. In other cases, however, it is possible to link a specific program to a specific outcome, 
and in these instances the relationship is clearly identified. 

Background 

This section includes a brief discussion of the local precedents and the theory of change 
for the SCALE project. 
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Origins of SCALE and Local Precedents 

When it began in 2003, the organizations partnering within SCALE were UW-Madison, 
the University of Pittsburgh, and four urban school districts (Denver Public Schools, Los 
Angeles Unified School District, Madison Metropolitan School District, and Providence Public 
Schools). CSUDH became a partner early in 2004. In the fall of 2004, a CSUDH math faculty 
member who was active in SCALE met with both a math and a science faculty member at CSUN 
to explore the prospect of SCAEE activities at CSUN. CSUN’s active involvement in SCAEE 
began in spring 2005 as faculty began designing professional development institutes for K-12 
teachers. 

As with many STEM education reforms, the SCAEE project at CSUN built upon 
previous efforts that had cultivated an experienced and motivated cohort of personnel, and 
established the procedural groundwork for future initiatives. As noted above, the LCU, TNE, and 
the CSP were particularly influential in these regards. In fact, a SCAEE participant noted that she 
had been approached by a colleague at CSUDH, who was actively involved in SCAEE, because 
of her previous experience with the CSP and other STEM education initiatives. Thus, without 
these previously existing and extant reform efforts, it is unlikely that CSUN would have been 
brought into the SCALE project in the first place. Once introduced, SCAEE was brought into the 
orbit of a community of STEM and education faculty who were already active in the CSP and 
TNE initiatives. These faculty became part of a larger cohort of CSUDH and CSUN faculty who 
were guided by of the SCALE leaders from UW-Madison and CSUDH. Apart from the CSUN 
provost, who had an advisory role in SCAEE, no CSUN people helped to lead the SCAEE 
project, in contrast to the other two IHEs (UW-Madison and CSUDH) included in the IHE case 
studies line of work. The significance of this leadership dynamic and its role in how SCAEE was 
designed, implemented, and perceived at CSUN should not be underestimated. 

Eurthermore, the SCALE project at CSUDH is closely aligned with the Quality Educator 
Development (QED) project (funded by the U.S. Department of Education’s Teacher Quality 
Enhancement Project), so that both SCALE and QED collaborated to design and implement 
these institutes at CSUN and elsewhere in the Los Angeles basin. Thus, it is necessary to take 
into account not only other CSUN-based reforms when evaluating the SCALE project, but also a 
CSUDH-based reform initiative that, in effect, served as the local conduit through which SCALE 
activities occurred. 

Theories of IHE Change 

SCALE is a systemic reform initiative that involves IHEs and K-12 partners to improve 
math and science teaching and learning through the entire educational spectrum. The SCAEE 
theory of change posits that to achieve lasting improvements in K-12 STEM education, the 
entire continuum of teacher training and professional development must be improved, with 
particular attention to improving the role IHE faculty play in designing and implementing 
preservice curricula and in-service programs. The SCALE theory of change is based on a 
systemic understanding of the educational systems that inform and support K-20 math and 
science education. The SCALE theory of change pertaining to IHEs holds that, if improvements 
in IHE participation in teacher preparation and professional development are to be sustainable 
and significant, then it is necessary that: 
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1 . STEM faculty improve their approaches to teaehing and learning; 

2. STEM and education faculty collaborate effectively to improve teacher preparation; and, 

3. headers at different levels of the institution work to overcome the conservative nature of the 
IHE by supporting faeulty partieipation in teaeher preparation. 

It also is important to eonsider the broader theory of change held by IHE actors, 
including SCALE leaders at UW-Madison and CSUDH. Their implicit theory of change was 
based on insiders’ understanding of the complex and loosely coupled properties of IHEs, and 
was foeused on “planting small seeds of change at points in the system deemed most likely to 
eventually yield large changes, and do so by building on and eollaborating with other change 
initiatives that complement SCALE goals, and by identifying and working with individuals 
already interested in these goals” (Hora & Millar, 2008). 

This approach is known as the “campaign approach to change,” which involves 
mobilizing people around a strategie theme that has staying power at a particular institution 
(Hirschom & May, 2000). A key strength of this approach is that the main actors are able to 
identify strategic opportunities for leveraging resources. These may include combining resources 
with other ehange efforts or institutions to aehieve like goals, or seizing an opportunity, sueh as 
the appointment of a sympathetie new departmental ehair or dean, to promote a reform agenda. 
This approach also entails a key challenge, which is that leaders must have a deep understanding 
of the institutions involved. Another challenge is that leaders may find it difficult to know if and 
when a projeet is meeting its own eriteria for success beeause goals, objectives, and strategies are 
not elearly stated prior to implementation. 

In contrast to UW-Madison and CSUDH, the campaign approach to change was not used 
to implement SCALE at CSUN; there were no CSUN leaders within SCALE, and the UW- 
Madison and CSUDH leaders did not have deep local expert knowledge of CSUN. Having 
neither a CSUN-based leader nor a theory of change speeifie to CSUN, SCALE, in effect planted 
its interventions at CSUN. In effect, at CSUN, UW-Madison and CSUDH SCALE leaders used 
an approach to institutional change designed for UW and CSUDH. Because it was CSUN faeulty 
who implemented this work, the operative theory of ehange at CSUN was less that of SCALE’S 
and more that of the ECU, TNE, and CSP projects and their implementers. 

Evaluation Criteria 

This section includes a brief discussion of the criteria used to determine program 
outcomes. Evaluators are increasingly recognizing that interventions in complex environments 
sueh as edueation, publie health, and researeh management often result in a variety of outeomes 
beyond those originally envisioned by the program designers (Patton, 2006). Researehers argue 
that a mechanistic worldview of evaluation, as embodied in the commonly used linear logic 
model, will “fail to identify many of the most important results of these processes” (Connick & 
Innes, 2001, p. 1). Accordingly, instead of focusing only on the outeomes as delineated by 
program designers at the outset of an intervention, evaluators need to view the context of the 
intervention as a complex system, in order to identify a wide array of possible outcomes (Patton, 
2006). This more context-sensitive approach was necessary due to the lack of measurable 
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objectives provided by SCALE leaders. It is also important to note formulating traditional 
evaluation measures (e.g., measurable objectives) does not require use of a linear logic model 
approach. However, the decision to formulate measurable objectives would strengthen context- 
sensitive evaluation designs that account for context, culture, and cognition. 

Using this approach, we found there is little precedent in the evaluation literature for 
categorizing types of outcomes that may result from an intervention in a complex environment 
such as CSUN. Lacking precedent, we chose to organize program outcomes according to their 
location in the ICE framework. In addition, we organized outcomes in terms of: 

1 . First-order outcomes, which were those that program implementers articulated and 
anticipated, and 

2. Second-order outcomes, which were those that program implementers did not anticipate. 

Changes to the institutional context most often emerge as second-order outcomes. 
Evidence for both types of outcomes is based on second-party evaluations, documentary 
evidence, or reports of changes by at least three respondents. In cases where only one or two 
respondents consider a particular change to instructional practice or behavior important, these 
changes are reported and the number of respondents claiming the change is specified. It is 
interesting to note that most of the first order changes appear in the external influences, internal 
structure, and practices categories, while most of the second order changes appear in the 
resources, pervasive values, and individual sense-making categories. 

SCALE Activities: March 2005 to August 2007 

SCAEE activities at CSUN have centered exclusively on summer professional 
development institutes for EAUSD K-12 teachers in math and science. During the period from 
March 2005 to August 2007, SCALE implemented the activities described here in detail. When 
outcomes of each activity are presented, we indicate the relevant category within the ICE with an 
italicized phrase. The first and second order outcomes, organized by the ICE categories, are 
summarized in Figure 2. Finally, since SCALE activities in southern California were conducted 
in close collaboration with the QED project (based at CSUDH), the following section refers to 
the projects jointly as SCALE/QED. 

Science Institutes 

Background. One of the primary goals of the SCAEE project was to develop high-quality 
professional development for K-12 teachers in the form of immersion units (SCAEE, 2003). An 
immersion unit is a carefully selected and designed learning opportunity in which students are 
engaged in the scientific inquiry process over an extended period of time (4 weeks), focusing 
intensely on a particular concept or big idea in the content area (Eauffer, 2004). Each immersion 
unit provides a coherent series of lessons designed to guide students in developing deep 
conceptual understanding that is aligned with key science concepts and the essential features of 
classroom inquiry specified in the state standards of the district for which each is designed. 
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In each unit, students learn academic content by working like scientists: making observations, 
asking questions, doing further investigations to explore and explain natural phenomena, and 
communicating results based on evidence. The immersion units were delivered to K-12 teachers 
through an intensive weeklong professional development session that took place during the 
summer break. 

Goals and objectives. In the original SCALE proposal, the overarching goal for the 
science immersion units was to “(D)evelop and implement immersion STEM learning 
experiences to ensure that every student in our partner districts experiences the process of 
engagement in an extended (e.g., 4-week) scientific investigation at least once a year (SCALE, 
2003).” The specific benchmarks for this goal included developing and piloting immersion 
projects at different grade levels, developing professional development materials for unit 
implementation, conducting professional development sessions across the partnering K-12 
districts, and facilitating the institutionalization of the units into district curricula. However, there 
were no measurable objectives assigned to these benchmarks for this activity. ^ Eurther, while 
CSUN respondents clearly articulated the larger science institute goals for IHE participation and 
outcomes, they were less specific about the objectives and strategies for achieving those goals. 
SCALE leaders indicated that these larger goals for IHE faculty inclusion are: 

1 . To have STEM faculty ensure that content is accurate, and to have education faculty ensure 
that the pedagogical methods are accurate; 

2. To engage all IHE faculty as learners and to impart a new understanding, appreciation, and 
experience with inquiry-based instructional methods; and 

3. To develop local capacity for professional development by “training the trainers.” 

However, absent measurable objectives related to IHE participation and outcomes 
pertaining to the science immersion unit activities, it is very difficult to assess if and how 
SCALE achieved its goals in this area. As noted above, were we using a linear logic model to 
evaluate the program, and focusing only on the outcomes formulated by the program designers, 
this situation would pose a problem. Because we instead are using a context-sensitive approach, 
we scanned a wide array of data on SCALE activities, including descriptions of the project’s 
implementation and participant data to identify outcomes, ranging from those specifically 
attributable to the program to changes in institutional context. 

How the institutional context influenced the activity. The science institute activity was 
influenced by the institutional context of CSUN in several ways (see Eigure 2). These are 
described below and referenced according to their location in the ICE. We note that, on the one 
hand, CSUN participants can be thought of as operating as independent agents in this activity, as 
they did not officially represent CSUN or their departments. On the other hand, because they 
brought to the science institutes perspectives that had been indelibly shaped by their experiences 
and positions at CSUN, and their participation in SCAEE was shaped, in part, by the structural 
features of their positions at CSUN, we view them in the context of CSUN. It is important to 



^ An evaluation of the QED projeet that foeused on aetivities at CSUDH may have artieulated measurable objeetives 
for the seienee institutes. 
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note that some faetors have a easeading effeet, sueh as national trends in edueational reform (la 
in Figure 2) leading to local pedagogical reforms (2j), which in turn cultivated a cohort of STEM 
educators at CSUN (3a). By linking the most immediately visible factors back to their origins, it 
becomes possible to identify which factors exert influences on discrete projects such as SCALE, 
and thus more accurately trace the precise nature of how complex systems interact. 

• Prior to SCALE. A first effect of the context on SCALE was the demanding workload (2h) 
which, combined with the academic calendar at CSUN, resulted in a limited pool of potential 
participants in the SCALE initiative. Many faculty respondents indicated that summer was 
the best time for them to conduct research activities, which some viewed as their core activity 
(2g, 4b). In addition, research productivity is highly valued and encouraged at CSUN as it 
brings in valuable external funding in a time of budgetary crisis (lb, 3c), elevates the prestige 
of individual faculty, their departments, and the entire university, and is an explicit goal of 
some CSUN administrators (2c, 3e, 5b). As a result, some faculty indicated that it was not 
feasible or desirable to commit to an entire week of SCALE activities (5a). A second 
influence of the institutional context on SCALE somewhat mitigated the first. A pre-existing 
cohort of STEM and education faculty who had been engaged in K-12 PD activities provided 
a ready-made pool of potential facilitators for the science institutes (la, 2j, 3a). These faculty 
had been involved in CSP and other long-standing activities for several years, and were 
viewed by many respondents as an easily identified cohort of STEM educators. A third 
influence was the prior experiences of these individuals, which deeply shaped their 
expectations and skill sets regarding K-12 PD and were thus introduced into the science 
institutes. Eor example, some STEM faculty participants had been had been engaged by the 
CSP primarily as a content expert in their field, and thus expected to participate in SCALE in 
the same fashion (6d). 

• During SCALE. The science institutes at CSUN were peripherally influenced by two 
contextual factors during the project’s implementation phase. First, the demanding workload 
continued to exert an influence on the faculty participants (2h). While they had successfully 
cleared or re-arranged their schedules to participate in the institutes, some still felt pressure 
from their other current or pending work obligations. While neither a surprise nor a situation 
unique to CSUN, this bears highlighting as an important backdrop for faculty as they engage 
in reform initiative such as SCALE. 

Second, other reform initiatives were simultaneously occurring at CSUN during the science 
institutes, including the provost’s ECU initiative, TNE, and the CSP (2j). While none of the 
initiatives were directly involved in the science institutes, each shared the support of the 
CSUN administration and to a certain degree, the aforementioned cohort of STEM and 
education faculty cohorts on campus. Thus, each activity had the potential to influence the 
other in myriad ways, including the diffusion of new ideas and the unintended consequences 
of one initiative’s activities spilling over into another (e.g., newly developed relationships or 
frustration with reform efforts). 

Activities. The activities related to the science institutes include the design of the 
immersion units, and then the actual summer institutes. This section includes a brief description 
of both sets of activities. 
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• Institute design process. The immersion unit design process involved interinstitutional teams 
who collaboratively developed grade-specific science immersion units, each of which were 
designed to address LAUSD science standards. During 2004-05, UW staff and other SCALE 
leaders intended to bring the CSU STEM and education faculty plus EAUSD science experts 
together to collaboratively design a high-quality unit, with the focus on K-12 teacher 
learning and instructional improvements. As a result of working on immersion units and 
modeling active learning pedagogies, UW staff and SCAEE leaders soon realized that they 
could also use this immersion in-service project as an opportunity to help STEM and 
education faculty improve their own approach to undergraduate teaching. 

What happened was that as we were developing the immersion units, [one UW staff 
person] came up to me and said, the most important aspect of this is not so much the 
product that we will prepare, the unit itself, but in the process of preparing it, the 
professional development that has occurred among the [IHE] faculty and the [K-12] 
teachers in working together to do this. We also realized that once we did the institutes, 
we needed [more] professional development for the professional developers. (SCAEE 
leader) 

As a result, the UW staff and SCALE leaders began to more explicitly develop the design 
process to engage all participants, including IHE faculty, as learners and practitioners. A key 
mechanism for incorporating faculty professional development into the design of the 
immersion units during 2005-06 was the leadership study group, which was comprised of 
representatives from UW, CSUN, CSUDH, and EAUSD. The goal of this group was to pool 
expertise and resources to design a high-quality professional development curriculum, and to 
collectively learn how to implement the unit for the upcoming summer institutes. By asking 
the leadership study group members to learn how to model the active-learning pedagogy 
embedded within the immersion units, SCALE institute leaders enabled a more intentionally 
professional development experience for the CSU faculty and EAUSD teachers. This 
experience included learning both core elements of subject specific pedagogical content 
knowledge and tricks of education, including classroom management. 

UW-Madison staff at WCER, CSUDH and CSUN STEM and education faculty, and EAUSD 
science experts and teachers collaborated to develop immersion units for the following four 
grades: 

• Eourth Grade: Rot it Right 

• Sixth Grade: Plate Tectonics 

• Seventh Grade: Variation and Natural Selection 

• Eighth Grade: Density and Buoyancy 

• Institute activities and participants. UW staff in collaboration with CSUN and CSUDH 
STEM and education faculty, and EAUSD science coordinators and teachers, co-facilitated 
five science institutes in 2006, and eight science institutes in 2007. Each institute was a 5 -day 
workshop. Eor the 2006 institutes 108 EAUSD teachers participated. Eacilitators included 
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nine LAUSD science experts or teachers, one CSUDH physics faculty, one CSUDH 
education faculty, two CSUN education faculty, one CSUN geology faculty, and two CSUN 
biology faculty. For the 2007 institutes, 162 LAUSD teachers participated. Data were not 
available for the affiliation of the 27 facilitators for the 2007 institutes. 

Outcomes: First-order. The first-order outcomes for this SCALE activity, and the 
category in which they are positioned in the ICF framework (see Figure 2), are as follows: 

• SCALE provided professional development workshops to LAUSD teachers (Id, Ig). In 
accordance with the primary goal of this activity, SCALE provided high-quality professional 
development workshops to 270 EAUSD science teachers."^ 

• SCALE engaged IHE faculty as learners (Ic, 2j, 6d). As previously noted, UW staff had 
hoped that by asking the leadership study group members to learn how to model the active- 
learning pedagogy embedded within the immersion units, the study group members would 
experience professional development themselves. This included learning core elements of 
subject-specific pedagogical content knowledge and tricks of education including classroom 
management. Several respondents reported that in their previous K-12 professional 
development experiences, they had been given only the role of content expert, which they 
performed in public lectures, study groups, or workshops. Eaculty who participated in the 
SCALE math and science institutes, particularly the members of the leadership study group, 
reported a vastly different experience. In fact, several faculty reported that they had to shift 
from their accustomed role of the content expert to a new role as a student of learning theory 
and inquiry-based pedagogies. 

I think I’ve gone from kind of the official content contact person to now working more 
with the philosophy of the immersion unit. So instead of automatically giving them [K- 
12 participants] an answer. I’m writing that on the left hand side of my notebook so that 
[I can identify] if some of the content questions are out of sequence for what we’re doing 
in the immersion unit. I write those down and save those questions so that when we come 
to a content piece or an activity that is more relevant to the question, then I can bring the 
question back and discuss it in terms of what our content activity is that time. [I learned 
that] through all of the facilitation practice and the PD development. (STEM faculty) 

This educational experience provided some STEM faculty participants the pedagogical 
training that they did not receive as doctoral students. Interestingly, this change was not 
limited to STEM faculty, as an education faculty participant also reported a shift in how they 
participated in PD. 



See Osthoff, E. SCALE Grade 6 Plate Teetonies seienee immersion units and professional development institutes: 
Design, delivery, and teaeher experienee. Also, other reports in this series inelude: Classroom observations of 
enaeted immersion lessons in Grade 6 LAUSD elassrooms: Impaets of an inquiry approaeh on lesson strueture, 
student engagement, eognitive demand, and student involvement, Teaeher Refleetions on first-time enaetment of the 
SCALE Grade 6 Plate Teetonies immersion unit: Supports, ehallenges, and effeets on elassroom instruetion and 
student learning experienee, SCALE Plate Teetonies Plate Teetonies seienee immersion unit implementation: 

Effeets on instruetional eontent in Grade 6 LAUSD elassrooms, and Effeets of implementation of the SCALE Plate 
Teetonies seienee immersion unit in LAUSD elassrooms on student aehievement on distriet assessments and student 
work on extended tasks. 
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I had a tendency in professional development to kind of pop in to the learner-teacher 
mode so I have to be very disciplined in terms of following the model that’s been 
developed. (COE faculty) 

• SCALE is giving faculty pedagogical tools to use in their own classes (2j, 6a, 6b). Based on 
self-report, SCALE is influencing individual faculty instructional practices through 
participation in the institutes. Yet without classroom observations or further investigation, it 
is impossible to assess the veracity of these self-reported changes, or the nature of any 
changes. However, it is clear that through participation in the math and science institutes, 
CSUN faculty are learning new pedagogical methods and tools that may be transferred to 
their instructional practices at CSUN. In some cases, this has meant specific pedagogical 
methods. For example, one faculty member used several of the problems from math institute, 
and also gained “well-grounded expectations” about how well students should be able to do, 
and a new confidence in leaving students to do more mathematical work like explaining and 
organizing solutions. For that faculty member, the experience amounted to “evolution, not 
revolution,” since he had been previously exposed to inquiry-based methods in math 
education. One life sciences faculty who had used the Fast Plants program in the past was 
considering adding an inquiry-based exercise, and hadn’t thought to do so until participating 
in the science institutes. In another case, UW staff noted that CSUN faculty reported using 
methods learned in the institutes. 

He was telling me that, “I used a think-pair-share in my class today.” I said “Oh yeah, on 
what?” And he said, “Well, it doesn’t matter on what, I used it, this is my first time, and it 
was great, they loved it. They were talking to each other and it was great.” So it was kind 
of groundbreaking for him and that was before he’d even facilitated an institute. I mean 
that was just from what he’d learned from reading the immersion unit and talking with us 
and planning. (UW staff) 

This direct transfer of institute-based methods to the CSUN classroom was corroborated by 
other faculty respondents. In other cases, institute participation has given CSUN faculty a 
new understanding of pedagogy. One respondent noted that the emphasis in the institutes on 
being transparent has helped her to become a better educator by encouraging her to examine 
pedagogy from multiple perspectives. Another respondent stated that the readings about 
pedagogical content knowledge spurred changes to pedagogical methods employed in IHE 
courses. In several cases, faculty reported that the institutional pressure to improve teaching 
practices, via the ECU initiative, helped make faculty more attuned to the potential 
applicability of new methods to their own courses. Faculty also noted that tools gained 
through immersion units directly address the mandate for student learning outcomes by the 
CSUN administration. 

[I have used] some of the assessment ideas to see if my students are actually beginning 
to take ownership of what I’m talking about during a lecture. That’s something that I 
hadn’t done in the past. I’ve been teaching for a while now and other than quizzes or 
exams, these assessments are very, very simple to determine understanding. It’s an easy 
way for me to check where I am, where they are, determine if my class, the majority of 
my class, understands key concepts that we’re talking about. (STEM Faculty) 
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Outcomes: Second-order. The second-order outcomes for this SCALE activity, and the 

category in which they are positioned in the ICE framework (see Eigure 2), are as follows: 

• SCALE is influencing other reform initiatives (2j). As previously noted, prior to SCAEE 
there were other similar initiatives active at CSUN, including the CSP. Since the small cohort 
of STEM and education faculty who have been engaged in the CSP for several years were 
largely the same group who participated in SCALE, there was a natural opportunity for 
cross-fertilization of ideas and innovation. In fact, some respondents indicated that their 
experiences in SCALE were leading directly to changes in how the CSP was designed and 
implemented. 

Another thing that’s been interesting is that [CSP facilitators] have transferred the 
Deborah Ball model with the yellow and blue eggs into [the CSP], so we’re carrying that 
beyond the SCALE immersion unit workshops. [Also] we’re trying to use the [SCALE 
facilitation] model where we do have co-facilitation since most of us at CSUN have been 
involved with SCALE, so we’ve transferred that model into the [CSP]. (STEM faculty) 

In addition, respondents viewed SCAEE and the CSP as being complementary initiatives, to 
the extent that K-12 participants were encouraged to enroll in both activities. 

hirst of all we encourage our teachers to do both projects, that they are different so they 
can get value out of doing both. And we did have overlap the last two years. We’ve had 
teachers that did both programs. (STEM faculty) 

• Further expanding and training the STEM education cohort (3a). An outcome related to the 
cross-fertilization among reform initiatives is the further growth and professional 
development of this small cohort of STEM and education faculty. SCALE contributed to the 
growth of this cohort by recruiting new faculty and instructors to become institute 
facilitators, including faculty from other IHEs (e.g., CSU Eos Angeles, El Camino 
Community College). By bringing new people into this small cohort, SCAEE played an 
important role in enlarging the professional resources available to this group, and increasing 
the likelihood for the cohort’s future sustainability. 

In addition, by engaging this cohort in an intensive professional development experience of 
their own, SCAEE infused new ideas and perspectives into this cohort. While some members 
of this cohort had been actively seeking out learning opportunities through other venues, the 
SCALE experience was described as a more concerted and intensive experience than others. 
Thus, SCAEE was a vehicle for taking some members of this cohort to the next level in terms 
of their knowledge of inquiry-based teaching and learning, and provided them with concrete 
tools that could be easily adapted to their own courses. 

• SCALE represents a more intensive type of interaction with K-12 than previously 
experienced (Ih, 6d). According to faculty respondents, the design process for the SCALE 
institutes represented a new type of collaboration with K-12 personnel. Where previously 
STEM faculty provided content expertise for professional development or outreach 
programs, they now were forced to model a new pedagogical approach that merged content 
and pedagogy. Where previously education faculty had mentored preservice teachers or 
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conducted research in K-12 venues, they too were plaeed into the unusual position of 
modeling a STEM-based aetive learning pedagogy. Another differenee from previous 
professional development experiences that a respondent observed was elose eollaboration 
with LAUSD scienee experts, with IHE and K-12 staff interacting as equal partners in 
designing and faeilitating the seienee institutes. 

One of the advantages to SCAEE for me has been the faet that Em aetually working side 
by side with people from LA Unified eentral offiees and scienee braneh. And though they 
have always supported our programs, they have not witnessed what we do and we 
haven’t witnessed what they do, and I thi nk that this has been a very good PR move for 
us in some ways and for them, beeause we are now in eommunication. (STEM Eaculty) 

This outeome is notable, as fostering interinstitutional partnerships is a eore goal of the MSP 
program. In this case, the new lines of eommunieation represent an important professional 
resouree for both the K-12 and IHE partieipants, similar to the previously mentioned 
networks that Dr. Oppenheimer had developed over the years with loeal K-12 distriets. Sinee 
his networks had proven to be important resources for younger faeulty, who took advantage 
of the goodwill and eontaets he had developed, these new relationships may result in similar 
resourees for future faeulty. 

Another effeet of these new relationships is the slow ehipping away of hierarehieal 
stereotypes that sometimes inhibit K-12/HE eollaborations. 

What happens in some of these eases is people [in K-12] look at you like “You’re a 
doetor, [you’re an expert],” and therefore they get defensive, thinking “Well, I don’t need 
somebody with a PhD to tell me how to teaeh,” Again, that’s a cultural thing and some of 
that is brought on by the [reputation] that some professors have with others, whieh I kind 
of resent. [In SCALE], that’s wiped out. In other words, we’ve now got to know eaeh 
other well enough for them to realize that we’re not coming in as higher ed faeulty to tell 
you how to do it, we’re eoming in to share and eollaborate with you. (STEM faeulty) 

Einally, this respondent also observed that this improved insight into “what’s going on in 
aetual elassrooms” has infiueneed her own preserviee teaehing and involvement in other K- 
12 PD aetivities. 

• Some faculty participants are resistant to engagement with institutes (6a, 6d). Systemie and 
individual barriers to sueeessful adoption of improved instruetional methods prevail at 
CSUN, despite the existenee of an institutional mission that values teaehing exeellenee, and 
extant reform initiatives like the ECU or TNE initiative. As a UW staff person noted, the first 
step in learning new methods of instruetion requires people to be fully eommitted, and 
“present in mind and spirit.” This respondent noted that some CSUN STEM and edueation 
faeulty who partieipated in the seienee institutes were not particularly engaged with the eo- 
faeilitation process, and thus did not suoeessfully learn how to model an aetive learning 
pedagogy for the K-12 teaehers. 

During one institute, a faeulty member was up at the front leeturing about inquiry. As I 
watehed, it was elear that he just didn’t get what we were doing here. As if what we were 
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training LAUSD teachers on had nothing to do with him. He seemed to know everything, 
theory-wise, about education. He was like a living textbook, and he delivered his PD just 
like that. It was almost like there was no thought on his end that this work applies to him, 
too. (UW staff) 

This respondent also emphasized the importance of self-reflection, where a learner is willing 
and able to assess the learner’s own instruction and critique it effectively. Again, some 
faculty were more amenable to this critical aspect of pedagogical improvement than others, 
and those whom some respondents considered not self-reflective included both STEM and 
education faculty. Another respondent noted that one participant, a part-time faculty member, 
was not actively engaged in the facilitation of the workshops and did not even participate as a 
content expert. According to this respondent, the faculty member instead graded participant 
coursework and generally “faded into the background.” The respondent also questioned why 
a part-time faculty member had been selected to participate if a goal of SCALE was to 
influence the STEM department itself Furthermore, the respondent noted that the faculty 
member was known to be antagonistic to constructivist pedagogies, which raised additional 
questions about their selection. 

SCALE/QED Math Institutes 

Background. The other primary SCALE activity at CSUN was a series of professional 
development workshops for LAUSD math teachers (junior high and high school). While the 
math institutes should be viewed as a joint SCALE and QED effort, the institute’s leaders and 
materials originally came from CSUDH’s Mathematics Department’s Center for Math and 
Science Education (CMSE). The CMSE has operated a CSUDH-based mathematics project for 
several years, for which faculty have designed and implemented mathematics professional 
development institutes for K-12 teachers. 

Goals and objectives. The goals of the math institutes were to increase student 
achievement in and understanding of the mathematics contained in the California state standards 
in Grades 6-9 through implementation of a professional development program, and to better 
equip teachers to lead their students to a deeper understanding of mathematics. The 
advertisement for the institutes also notes that the designers hope to develop a core community 
of K-12 leaders to become resources for other educators. Furthermore, according to a 
respondent, a goal of the pre-algebra institute was to de-politicize math education by focusing 
more on the commonly held goal of improving student learning, and less on the ideological 
and/or political aspects of math curriculum. 

How institutional context influenced the activity. The math institute activity was 
influenced by the institutional context of CSUN in several ways, which are described below and 
referenced according to their location in the ICE (see Figure 1). 

• Prior to SCALE. First, the ideological schism that has long prevailed in the CSUN math 
department (2i) served as an important and unshakeable backdrop to the SCALE math 
institutes, and effectively limited the pool of potential facilitators. The department is split 
into two camps with divergent perspectives on math education, one of which is, and one of 
which is not sympathetic towards a constructivist approach to math education. Members of 
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the former would be the most likely reeruits to faeilitate the SCALE institutes. Indeed, one 
COE respondent noted that this group is well known to most edueation faculty to the extent 
that they simply ignore those mathematicians who are antagonistic towards their efforts. 
Second, some members of this cohort were either tenured or close to retirement, which 
afforded them a certain degree of professional security and insulation from the critiques of 
their colleagues (3e, 5b). These faculty would also not be subjected to potential 
recriminations or retributions through venues such as RTF committees. 

• During SCALE. The split in the math department played a direct role in the 2006 math 
institutes, as one of the facilitators proved to be somewhat antagonistic to the intent of math 
education reform in general, and the design of the math institutes in particular (2j). This 
particular facilitator had been selected by the department chair and not by the SCALE leaders 
from CSUDH or UW-Madison. Another factor influencing the math institutes was the TNE 
initiative, in which some math department faculty were already engaged prior to SCAEE (2j). 
As a result of the simultaneous initiatives, information and ideas from one influenced the 
other. 

Activities. In the summers of 2006 and 2007, SCALE sponsored math institutes for 
middle school (pre-algebra) and high school (algebra) teachers. According to respondents, the 
two math institutes were designed in slightly different ways. In 2006, the pre-algebra institute 
was largely modeled after a similar workshop used at CSUDH, while the algebra institute was 
loosely based on a CSUDH workshop but with significant modifications made by the facilitators. 
The institute design process for the algebra institute was described by one respondent as 
problematic, with insufficient time to work with co-facilitators, and minimal guidance regarding 
unit structure, sequencing of activities, and the actual content of the institute. In 2007, the 
facilitators had a 4-day training led by CSUDH personnel, which included analyses of 
videotaped PD sessions. Respondents noted that this training, combined with more time for 
planning and experience from the previous year, led to a more satisfactory and coherent series of 
workshops. 

The SCAEE/QED mathematics institute strategies are to use an inquiry-based 
methodology while focusing on the LAUSD mathematics curricula and instructional guides. 
During each of these 120-hour institutes, the teachers work with materials specially selected to 
increase the algebraic thinking and problem solving capacities of the teachers in order to help 
them develop their own mathematical explanation structures. According to official 
advertisements, these institutes included unit development and lesson planning, discussions of 
current research addressing English language development (EED) and math issues, and 
explorations of assessment methods that could inform instructional practice. 

Cross-institutional teams including CSUN STEM faculty, MMSD personnel, and 
LAUSD personnel then co-facilitated 15-day math institutes (2 in 2006, 2 in 2007) for a total of 
83 LAUSD teacher participants. 

Outcomes: First-order. The first-order outcomes for this SCALE activity, and the 
category in which they are positioned in the ICE framework, are as follows: 
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• SCALE provided professional development workshops to LAUSD teachers (Id, Ig). In 
accordance with the primary goal of this aetivity, SCALE provided professional development 
workshops to 83 LAUSD math teachers. Available evaluation data did not address the effeets 
of the workshops on partieipant knowledge gains or other similar outcomes. 

Outcomes: Second-order. The seeond-order outeomes for this SCALE activity, and the 
category in whieh they are positioned in the ICE framework, are as follows: 

• SCALE complemented the efforts of other reform initiatives (2j). As previously noted, prior 
to SCALE there were other similar initiatives aetive at CSUN, including TNE. Since the 
small eohort of STEM and education faculty who have been engaged in TNE were largely 
the same group who participated in SCALE, there was a natural opportunity for eross- 
fertilization of ideas and innovation. In faet, some respondents viewed their dual partieipation 
in SCALE and TNE as complementary. 

I don’t think I ean separate [the effects of SCALE or TNE]. I think that this is sort of an 
ongoing sort of professional evolution for me. TNE’s foeus is what we do here in the 
university, while SCALE is mueh more about trying to eonneet with teaehers and 
influenee in-service [training]. So why neglect one or the other? It seems clearly the case 
that both are interesting. (STEM faculty) 

• SCALE helped to expand and train the cohort of math educators (3a). An unanticipated 
outeome related to the math institutes was the further growth and professional development 
of the small eohort of math educators in the math department. SCALE contributed in a small 
way to the growth of this eohort by reeruiting a junior faculty and an adjunct faculty member 
to become institute faeilitators. However, as previously noted, one of these facilitators was 
generally not supportive of the rationale behind the institutes, and reportedly had an 
unsatisfactory experience. In addition, by engaging this eohort in an intensive professional 
development experienee of their own, SCALE infused new ideas and perspectives into this 
cohort. While some members of this eohort had been actively seeking out learning 
opportunities through other venues, such as TNE, the SCALE experienee provided an 
opportunity for faculty to test out new ideas and integrate their knowledge regarding active 
learning strategies and student assessment. 

One of the cornerstones of the SCALE grant is getting teaehers to do things like unit 
planning that will help eneourage them to include what they’ve learned in their own 
teaching. [In 2006], we recognized the importance of it but I thi nk we felt somewhat 
unfamiliar with and uneonneeted with the various forms and materials that they [CSUDH 
faeulty] were using. So this year [2007] we tried to do it ourselves and we did something 
that I think was suceessful in various ways. We looked through the math instruetional 
guides and ehose concept units, and particular topics that are in the focal points issued by 
the NCTM and tried to have teachers refashion these units, build plans that would sort of 
lead them to go through these units in a more reflective way. And so a lot of our institute 
was based on problem-solving and explanation (STEM faculty). 
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Thus, SCALE was a vehicle for taking some members of this cohort to the next level in terms 
of their knowledge of inquiry-based teaching and learning, and provided them with concrete 
tools that could be easily adapted to their own courses. 

• SCALE became involved in the math department schism (2i). One unanticipated SCALE 
outcome was involvement in the math department schism. In 2007, Dr. David Klein, a 
member of the department who has a national reputation as a strong opponent of NSE-funded 
math education reforms, wrote a letter to the editor of the American Journal of Physics 
critiquing the SCALE project. The principal investigator (PI) of SCAEE wrote a response 
that was published, and as a result, the SCALE project became actively involved in the math 
department schism. 

V, Exploring How Culture Mediates and Influences STEM Education Reform 

As previously noted, some pervasive values and beliefs at CSUN posed a significant 
challenge to affecting the types of institutional and pedagogical changes envisioned by the 
SCAEE project. As a result, we now turn to a detailed analysis of how cultural factors mediate 
and influence STEM education reform. By taking this fine-grained approach to studying culture, 
we hope to articulate the specific content and processes comprising cultural dynamics at CSUN, 
as opposed to relying on generalizations about the institution’s culture. In this section, we 
analyze the mental model of one CSUN faculty, how it is situated in the institutional context of 
CSUN, if and how STEM education reforms interacted with the mental model, and effects of 
STEM education reforms, if any, on the mental model. 

Using a Distributed Theory of Cultural Analysis From Cognitive Anthropology 
Background 

This focus on sense-making is inspired by research on reform implementation in both 
higher education and K-12, which has found that policy directives are frequently adapted and 
transformed by individual agents at the local school or IHE level. The process of interpreting 
policy interventions and adapting them to one’s own local situation is sometimes called sense- 
making, where institutional actors “make sense” of their environment and select appropriate 
actions (Bimbaum, 1988). Increasingly, researchers are recognizing the importance of this 
process and of how local contextual factors such as organizational structure and leadership shape 
the sense-making process (Coburn, 2001; Spillane, Reiser, & Reimer, 2002). These issues are 
important due to the oft-cited slow rate of change in IHEs and the poor fidelity of policy 
interventions to the undergraduate STEM classroom. A focus on instructor sense-making is an 
important departure from the popular view that external pressures (e.g., MSP program) are de- 
coupled from the classroom, which operates in an autonomous vacuum immune from these 
exogenous forces (Cobum, 2001). Instead, as we have seen, forces such as doctoral training 
programs at research universities are introduced into a new institutional context, where they are 
in turn subjected to local conditions and reinforced, rejected, or assimilated. 

Of particular interest to sense-making theorists and STEM education reformers is the role 
that the constellation of pervasive values and behavioral norms collectively held by groups 
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within an institution (commonly known as institutional or organizational culture) plays in 
shaping individuals’ sense-making proeesses, and ultimately, their behaviors and interpersonal 
relationships (Becher & Trowler, 2004; Van Maanen & Barley, 1984). If it is true that values, 
beliefs, and tacit assumptions play an important role in reform implementation, then it is 
extremely important to understand preeisely how culture manifests during the implementation 
process, and if it influences the observed outeomes of the reform. Further, as this ease study 
demonstrates, pervasive values and beliefs do not exist in a vacuum within organizations, but 
instead are shaped and influenced by the technical and social constraints of the institutional 
eontext. Thus, a eultural account of the implementation proeess must also attend to the recursive 
relationship between eulture and the institutional context. 

The challenge of empirieally situating individual sense-making into complex institutional 
and eultural contexts is inadequately met by the dominant approach to understanding and 
analyzing organizational culture, whieh produees statie and homogenous sets of vaguely defined 
values, beliefs, or artifacts (Bate, 1997; Van Maanen & Barley, 1984; Knight & Trowler, 2000). 
In partieular, many studies fail to address the degree to which group members adhere (or not) to 
the dominant knowledge in their organization (Ross, 2004). Moreover, some studies of 
organizations ignore the role of time in eultural proeess, thereby obseuring the origins and 
evolution of eommonly held beliefs and values and, ultimately, our understanding of the actual 
processes of cultural change within organizations (Hudelson, 1997). 

Complicating the methodologieal challenges of measuring the sense-making process and 
its contextual determinants is the interdiseiplinary nature of the topic, and the subsequent 
panoply of theories, methods, and analytic procedures currently in use. For example, researchers 
are addressing this topic by studying decision-making processes (Coburn, 2005), eausal mapping 
(Shavelson, Ruiz-Primo, & Wiley, 2005), organizational learning (Akgun, Lynn, & Byrne, 

2003), and mental model composition (Webber, Chen, Payne, Marsh, & Zaeearo, 2000). For the 
purposes of this study, we have drawn on eognitive anthropology to develop an approaeh to 
study the roles that culture, context, and cognition (i.e., our interpretation of sense-making) 
played in the implementation and ultimate outcomes of the SCALE project at CSUN.^ 

A New Approach: Situated Mental Models Theory 

In contrast to a commonly used theory of culture that seeks to establish a unitary and 
stable set of eultural norms for a specific group of people (e.g., the math department), we employ 
here a distributed theory of culture. This approach accounts for cultural phenomenon at the 
group and the individual levels by eoneeptualizing culture as knowledge and practice that is 
distributed aeross and within groups, and is also internalized by individual members of a group 
(Shore, 1996; Atran, Medin, & Ross, 2005). Furthermore, aeeording to this approach culture is 
not simply the aggregate of individual cognition, but is comprised of the eonstellation of 
knowledge and beliefs held by group members that is also reified, reinforeed, and reproduced by 
soeial action and instantiated in artifaets and other eultural forms (Shore, 1996). In this way. 



^ In the future, we will thoroughly review these various literatures to ensure that the most appropriate and valid 
methods are used to study STEM edueation reforms. 
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some cognitive anthropologists are striving to maintain a dual focus on the importance of the 
individual and the presence of a group-level culture (Quinn, 2005). 

The idea that an individual’s cognitive processes shape how information (e.g., visual or 
aural stimuli) is processed and interpreted is an old concept in psychology and the cognitive 
sciences (Rumelhart, 1980; Schank & Abelson, 1977). Schema theory is one of the core ideas in 
cognitive psychology which posits that incoming information must be reduced in its specificity 
(e.g., color, context) and chunked into more simplified units that can be stored in short term 
memory. The schema that individuals internalize constitute “unconscious mental structures that 
underlie molar aspects of knowledge and skiU” (Brewer, 1987, p. 188). As individuals internalize 
information and experiences from their physical and socio-cultural environment, or schematize 
them, they become deeply embedded in the cognitive processes of the brain through repetition, 
reinforcement, and attachment to key life events or emotions (Shore, 1996). In addition, schema 
tend to be domain-specific (e.g., plate tectonics) and may be attached to specific stimuli or 
environmental conditions. 

Importantly, cognitive psychologists distinguish between schema as underlying 
knowledge structures, and their episodic representations, generally called mental models, which 
are formed in active engagement with environmental stimuli (Brewer, 1987).^ These episodic 
representations are more complex “global knowledge structures constructed at the time of input” 
that may be comprised of various interlocking schema that collectively comprise the explanatory 
structures that actors use to filter (by omitting or transforming) environmental stimuli (Brewer, 
1987, p. 189). Further, there are different types of mental models that serve different purposes 
for individuals as they interact with their technical, social, and cultural surroundings (Shore, 
1996). For example, there are orientational (e.g., spatial models, social orientation models), 
diagnostic (e.g., checklists), conceptual (e.g., theories, classificatory models), and task models 
(e.g., scripts for specific task completion). As with individual schema, mental models may also 
become associated with specific stimuli or environmental conditions. 

Some cognitive anthropologists also suggest that mental models are comprised of 
combinations of different types of schema (D’Andrade, 1995; Strauss & Quinn, 1998). Some 
schema can be considered cultural since they are internalized from instantiated cultural forms 
that are “part of the stock of shared cognitive resources of a community” such as the Star- 
Spangled Banner or tacit food habits pertaining to specific mealtimes and holidays (Shore, 1996, 
p. 47).^ Thus, individual mental models are comprised of a variety of schema, some of which can 
be explicitly linked to instantiated cultural forms (e.g., rituals, artifacts) or socially sanctioned 
beliefs and practices for a specific group. In addition, cultural schema may be internalized 
differently by particular members of the group, and may be tacitly held and never actually 
surface in everyday interaction. It is also possible that an individual may internalize the cultural 



^ The use of the term “mental model” is ubiquitous and has been eritiqued for its laek of eoneeptual elarity and 
definition, partieularly in organizational seienee (Lakomski, 2002). The term is more preeisely defined in eognitive 
psyehology, espeeially in its relationship to sehema. However, the eonstruet still suffers due to laek of attention to 
variability, ehange, and distinetions between eonstituent sehema that are idiosyneratie vs. internalized from eultural 
forms. We use the term here as a bridge to diseussing different types of sehema and the eultural model eonstruet. 

’ It is important to note that by labeling a speeifie sehema as “eultural,” we am not implying that the mind does not 
mediate all historieal, soeial, and eultural environments (Vygotsky, 1978; Wertseh, 1991). 
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schema from a variety of different groups of whieh they are members (e.g., ehurch, department, 
family). 

In organizational life, an individual’s mental model and its constituent schema are 
imported into a new institution, and over time certain models may adapt to the unique policies, 
structures, soeial groups, and behavioral norms extant in a given organizational unit (Lakomski, 
2001). Once situated into the environmental context of a workplace, these mental models are 
activated in response to environmental stimuli, such as a new syllabi or a policy intervention, 
sueh as the MSP program. However, mental models and schema are not immune to ehange or 
evolution, and may be altered in response to new information or shifts in the environmental 
context of the workplace. In addition, individual cognition and practice may create or reinforce 
core elements of this environmental context in a recursive relationship (Giddens, 1984). 
Researehers have long argued that over time multiple individuals with similar mental models can 
alter institutional structures and traditions (Clark, 1998). 

As a result, mental models must be viewed as simultaneously situated within and 
influenced by the organizational context (organized in terms of the ICF in this study), and shaped 
by the individual’s unique personal experiences and social position in that context. In sum, we 
believe that this approach — ^by accounting for how individuals variously interpret, internalize, 
and instantiate organizational values, attitudes and knowledge — enables evaluators and 
researchers to develop a more adequate understanding of organizational culture than an approach 
that presents knowledge as uniformly shared and distributed aeross a eomplex organization. We 
posit that this distributed approach to culture is partieularly suitable for evaluating systemie 
reforms (like the ones that SCALE is attempting in higher education) for two reasons: (a) Faculty 
are particularly autonomous agents who exhibit high degrees of variability and independence and 
thus may hold cultural schema that are not evenly shared aeross their departments and other 
groups; and (b) SCALE is attempting to introduee into the environment new eultural schema that 
individual faculty may internalize in various ways. We hope this provides an approach to the 
study of organizational culture in IHEs that is more nuanced and empirieally sound and thus of 
value to the STEM education community. 

A Note on Measuring Situated Mental Models 

The question of how mental models and their constituent parts are instantiated into an 
empirically observable form is a critical issue. For some researchers working in laboratory 
settings, schema or other aspects of cognition can be observed through experiments involving 
research subjects making causal judgments or comparing subjects’ recall of critical events over 
time.^ It is considered more challenging to observe eognitive proeesses in non-laboratory 
settings, as there are more variables to account for and thus less control over the stimuli affecting 
a research subject. Some researchers have focused on highly standardized workplace 
environments in order to analyze how the mental models required for eompleting a task, sueh as 
navigating a battleship, can be de-centralized among a group’s members and how eritieal 
information can be embedded within artifacts (Hutchins, 1995). However, for researchers 
interested in cognitive proeesses in non-standardized environments where cognitive processes 



^ These are only two seleeted examples of lab-based researeh on eognition. 
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may not be easily instantiated or embedded in physieal form, the role of discourse and narrative 
plays an important role (Quinn, 2005). This is a critical issue for researchers in management, 
education, and industry who wish to understand how cognitive processes unfold in complex 
environments (Lakomski, 2002). 

Researchers seeking to empirically observe cognitive processes commonly analyze 
dialogue in interview transcripts (Brewer, 1987; Quinn, 2005). In addition, sense-making 
theorists have long used structured interviews to elicit respondents’ decision making and 
interpretive processes (Dervin, 1998). In the field of cultural anthropology, researchers have 
been using surveys (e.g., paired comparison, freelist), participant observation, and in-depth 
interviews to explore the cultural components of individual sense-making and knowledge (Atran, 
Medin & Ross, 2005; Quinn, 2005). As our data for this evaluation were limited to interviews, 
we chose to work within the interpretive tradition of Strauss and Quinn (1998) that uses natural 
discourse to identify the cultural aspects of individual cognition. 

For this research, we have focused on individuals’ statements of fact (e.g., tacit 
assumptions), values, or beliefs about STEM education reform as evidence for the schema that 
comprise their mental models. Individual schema were considered cultural only if they met two 
criterion: (a) the schema was expressed by at least three individuals within a specific social 
group at CSUN (e.g., entire university population, college, department), and (b) the schema was 
explicitly linked to this specific social group by at least one respondent and/or observation-based 
evidence. As a result, this study does not attempt to present an account of the culture of an entire 
organization or even its administrative sub-units, but instead presents the pervasive cultural 
schema held by group members in a specific domain under consideration. 

We then build on and extend this approach to cultural model analysis by explicitly 
linking mental models and cultural schema to specific factors in the local institutional context, 
and by attempting to develop knowledge that can guide local program planners and 
policymakers. This is done by drawing on the causal network analysis method, which is “an 
abstracted, inferential picture organizing field study in a coherent way” (Miles & Huberman, 
1994, p. 153). The relationships between individual schema and contextual factors were either 
explicitly stated by respondents, or formulated through analytic inference. In addition, we draw 
on the theoretical perspective of Bourdieu (1977) in order to interpret the relationship among 
individual disposition (i.e., habitus), capital, and the organizational field (i.e., the institutional 
context(s). We strongly emphasize that this is an exploratory effort to identify situated mental 
models, undertaken in order to improve the understanding of the role of culture and the 
institutional context in STEM education.'^ (See the Appendix, for more details.) 



® Future research should be undertaken to build upon this exploratory effort. This research may involve the 
additional use of methods developed specifically for identifying agreement, such as freelisting and cultural 
consensus analysis, For the analyses presented here, we identified cultural models using data from two code 
categories in the ICF: collective values, where respondents were prompted to speak about their group’s tacit 
assumptions, and individual sense-making, where respondents expressed personal perspectives. 
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Analyzing Situated Mental Models in Action 

In this section, we describe the situated mental model of a CSUN faculty member prior to 
the SCALE project, how the model interacted with STEM education reforms, and its 
composition at the conclusion of SCALE. The faculty member is in a STEM department, and did 
not participate in SCALE. The analysis focuses on the following points: 

1 . The cultural schema for STEM education reform at CSUN, based on the pervasive values 
identified earlier in this evaluation; 

2. The primary contextual factors that shape individual mental models and impinge upon the 
sense-making process; and 

3. The mental models (and salient contextual factors) of the faculty prior to, during, and after 
the SCAEE intervention. 

Eor this analysis, the pervasive values and individual sense-making processes of interest 
are limited to the domain of STEM education reform, which at CSUN includes not only SCALE 
but also the ECU, TNE, and CSP initiatives. The presence of multiple reform initiatives was an 
unpredictable boon for this research, as they laid the cognitive groundwork in advance such that 
individuals interpreted the SCALE project through existing mental models that had been 
developed over time. Many CSUN faculty have had ample opportunities to form opinions and 
solidify perspectives about reform, and thus did not create interpretive frameworks “on the fly.” 
Thus, CSUN faculty generally exhibited a well-formulated set of schema, and were sufficiently 
self-reflective about their sense-making processes that we did not have to dig very deeply for 
these data during the interview process. 

The Cultural Schema Pertaining to STEM Education Reform at CSUN 

As previously noted, during the evaluation of the SCAEE project we identified four 
pervasive values and beliefs pertaining to STEM education reform. These values and beliefs 
were expressed by individuals in a variety of administrative units, disciplines, career stages, and 
levels of participation in STEM education reform, and meet the criterion for a cultural schema. 
Thus, these cultural schema may be considered salient aspects of the institutional context of 
CSUN that transcend specific social groups or administrative units. To review, the pervasive 
beliefs and values identified in this evaluation are as follows: 

Scientific legitimacy and credibility equated with basic research accomplishments. 

Since no STEM respondents had prior training in pedagogy in their graduate training, this clearly 
conveys the higher value placed on research activities and on acquiring expertise in a specialty 
area of their field. The importance of achieving legitimacy and credibility in a particular 
discipline is viewed as a critical ingredient of academic success in general, and for 
interdepartmental collaborations in particular. Competence in a discipline and specialty area is 
made legitimate and credible by a faculty position, publications, and an active research program. 
Moreover, legitimacy in a STEM discipline often is more highly valued than in education. 
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Distinction between hard and soft sciences fuels long-standing tension between 
departments in the school of education and some STEM departments. A corollary of the 
faculty members’ close professional identities with their disciplinary traditions is that different 
faculty may hold value systems that differentiate disciplines in a hierarchical order. In particular, 
academics commonly hold a schema that distinguishes between the hard and soft scientists, 
where the hard sciences are considered more rigorous, as they take a scientific approach based on 
reproducible experimental data, and the soft sciences are considered more interpretive, anecdotal, 
and less scientific. Using this schema, STEM faculty generally are identified with the hard 
sciences, while education faculty are identified with the soft sciences. 

Tension between institutional support for reform and the disciplines. While the 
predominance of the disciplinary value of research accomplishments serves to inhibit the goals 
of SCALE, the teaching oriented mission of CSUN serves to support SCALE. The resulting 
tension has important implications for inter-departmental collaborations and for dictating how 
individual faculty prioritize their workload, both of which are of critical importance to the 
SCALE project. 

Beliefs about the relative importance of content and pedagogy in preservice STEM 
courses. Respondents made clear that within certain departments there are major disagreements 
about the courses required to effectively train a K-12 teacher, and the approach that the 
university should take in designing and offering these courses. Some faculty adopted the position 
that preservice courses should focus primarily on covering the canon of their discipline, while 
others emphasized the importance of pedagogical concerns (e.g., lesson planning, scaffolding, 
active learning strategies). This particular schema was also linked to an individual’s level of 
exposure to the learning sciences in graduate school or in previous jobs, with those who had little 
to no exposure tending to feel that content coverage was of utmost importance to preservice 
training. 

Again, it is important to note that we do not claim that all faculty, administrators, and 
staff at CSUN exhibit these values or beliefs. Instead, while these values were expressed by 
several of respondents across different social groups and administrative units, individuals may 
differentially internalize and exhibit these cultural schema depending on their personal 
backgrounds, personality, social position, discipline, and many other situational or demographic 
variables. 

Primary Contextual Factors That Shape Individual Mental Models and Impinge Upon the 
Sense-Making Process 

As previously noted, our conceptualization of mental models includes both the schema 
that comprise the cognitive processes of an individual (i.e., cultural and other schema), and the 
contextual determinants that shape and influence the complexion of these schema and the 
subsequent sense-making process. Thus, it is important to identify the specific contextual factors 
that impinge upon the sense-making process and activate an individual’s mental model. Based on 
the data from this case study, we speculate that when faced with a STEM education reform, the 
following four contextual factors (listed in order of importance) exert the most influence on 
individual sense-making processes: (a) workload and attendant RTF considerations; (b) 
availability of fiscal resources, (c) availability of social resources, and (d) leadership. 
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Of course, the real-time interpretation of a situation is inextrieably linked to the unique 
personal eharaeteristies of the individual, partieularly their personality and personal eonneetion 
to the topie under eonsideration. For example, if a faeulty member has a ehild in a loeal K-12 
distriet, that person may be more receptive to or critieal of a new STEM edueation reform effort. 
Other individual-level faetors inelude personal baekground, ethnieity, soeial status, and 
disciplinary training. Thus, the eultural sehema and eontextual faetors identified in this analysis 
interaet with these idiosyncratie features in a dynamie relationship. Next, we turn to how one 
faeulty member internalized the eultural sehema at CSUN, and the unique eharaeter of these 
sehema prior to, during, and after the SCALE intervention. 

The Mental Model of a STEM Faculty (Non-SCALE Participant) for STEM Education 
Reform 

The following seetion ineludes a description of the eontextual faetors most salient to this 
respondent’s mental model for STEM education reform, and a description of the respondent’s 
aetual mental model prior to, during, and after the SCALE intervention. This partieular 
respondent did not partieipate in SCALE, and the “during reform” phase refers to the TNE effort 
undertaken in the respondent’s department. Figure 3 depiets the processes deseribed below. 




Note: Only a few connections between elements of this model are noted for 
illustrative purposes 



Response is uniquely 
expressed depending 
on the individuals 
mental models and 
their relation to the 
political, social, and 
historic environment. 

Response: 

Two applicants are 
interviewed and 
considered not 
suitable. 
Respondent 
indicates that the 
experience may have 
hardened positions 
on STEM education 
reform in the 
department. 



Figure 3. A faculty member’s mental model for STEM education reform. 
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This particular respondent was hired relatively recently at CSUN, and it was the 
respondent’s first academic appointment. Thus, the primary contextual factor that shaped the 
respondent’s mental model for STEM education reform was that of workload and attendant RTF 
eonsiderations. Due to this coneern, the approval of peers in the department was partieularly 
important, whieh made social factors in the college and department partieularly influential. 
Finally, the influence of the respondent’s doctoral training was still fresh and deeply shaped 
perspective on certain topics pertaining to reform, pedagogy, and disciplinary identity. The 
following analysis includes a description of this faculty’s mental model, and its constituent 
schema, for STEM education reform prior to, during, and after the SCALE intervention. 

Within STEM disciplines, research is given priority over teaching and service (CSUN- 
wide cultural schema). Responses are presented pre-, during, and post-reform. 

• Pre-reform (Tl). As noted above, this respondent was hired relatively recently at CSUN. As 
a result, the respondent was almost singularly focused on conducting research, publishing 
research articles, raising external research funds, and teaching the obligatory 12 units per 
semester. These immediate and praetieal coneerns about time management were related to 
the eultural schema that prioritizes researeh over teaching and service aetivities. This sehema 
makes it unlikely that a new STEM hire would participate in a reform effort, as it would 
impede progress towards tenure and promotion. 

This eultural schema was expressed by this respondent as a taken-for-granted perspeetive on 
academic life in a STEM discipline. As a doctoral student in a major research university, the 
respondent had never been in an institutional environment where the prioritization of 
research over teaching and service activities was not the case. Furthermore, this schema was 
reinforced locally at CSUN by the department’s RTF policies and CSUN leadership. When 
new faculty are hired in this department, they sign a memorandum of understanding (MOU) 
that specifies the expectation that they will publish at least three research papers prior to their 
RTF review. There are no corresponding details about expectations regarding teaching or 
service activities. Furthermore, the respondent noted that some CSUN leaders have eonveyed 
explicit messages regarding the expeetations for faeulty researeh produetivity, ineluding the 
statement at a public meeting that junior STEM faculty “should be living in the lab.” 

• During reform. The SCALE projeet and other reform efforts at CSUN did not directly 
activate this schema for this respondent. However, since the TNE effort did engage the 
department in an attempt to hire a pedagogy specialist, this respondent’s deep-seated notion 
about the prioritization of duties was engaged. Sinee researeh was eonsidered the top priority 
in the department, the respondent felt that, while the TNE position “would be nice,” it would 
essentially be useless since “(I)nvesting time in becoming a good teacher is not rewarded and 
we are not really incentivized to spend inordinate amounts of time developing new teaching 
tools.” Thus, the schema for the prioritization of researeh rendered this reform attempt 
superfluous and actively at odds with the prevailing value system of the department. 

• Post-reform (T2). There were no changes to this schema as a result of the SCALE project and 
other reform efforts. In fact, the failure of the TNE effort to hire a pedagogy specialist may 
have reinforeed this partieular sehema by demonstrating that a researeh-eentered perspeetive 
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is truly the dominant one in the department, in spite of the allure of a fully funded tenure- 
traek appointment. 

Beliefs about the relative importance of content and pedagogy in preservice STEM 

courses (CSUN-wide cultural schema). Responses are presented pre-, during, and post-reform. 

• Pre-reform (Tl). This respondent had neither taught prior to aeeepting a position at CSUN, 
nor taken any professional development eourses in teaehing and learning. While a graduate 
advisor had once experimented with inquiry-based teaching in a lab, and found that “there 
must be something going on since they are all so engaged,” this experience was trivial 
compared to the focus on research accomplishments and the mastery of disciplinary content. 
Thus, the respondent’s schema for the relative importance of content and pedagogy in 
undergraduate STEM courses was almost without content with regard to pedagogy. Because 
this respondent did not think about pedagogy, the schema for this topic was not about the 
relative importance of content or pedagogy, but rather about content. Furthermore, as the 
respondent was not aware of any preservice teachers in the respondent’s courses, the 
respondent had essentially no opinions or beliefs about which approach would be best for 
that student population. 

However, despite lack of experience and training in education, this respondent had strong 
beliefs about the factors that led to poor student performance in the STEM fields at CSUN; 

(a) since most students were non-traditional (e.g., part-time students, parents, English 
language learners), they did not have the time or energy to adequately devote to the study of 
STEM fields, and (b) the lack of parental involvement at a young age led to many younger 
students developing the poor work habits shown by some undergraduate students. Thus, in 
the absence of a coherent position on this particular schema (which is itself an internalization 
of the discipline’s perspective on this topic), this respondent had developed strong personal 
beliefs about student learning that have little overlap with findings developed by the learning 
sciences. In other words, this respondent held a type of folk theory about student learning. 

• During reform. This schema was directly engaged by the TNE effort to hire a pedagogy 
specialist in the department. First, a candidate for the TNE position did a teaching 
demonstration and chose an inquiry-based assignment for an introductory course. The 
respondent noted that the lesson was “a really interesting way of going about it, which forced 
[you] to confront what you don’t understand first, and then see facts to back up how to get 
the answer.” While the respondent noted “[I could] see how it would be much more effective 
than just the passive lecture approach,” the respondent also agreed with the reaction of a 
senior faculty member who observed that “he just spent 45 minutes covering something that 
could have taken 10 in a lecture.” The respondent deemed that taking this much time to cover 
a relatively small amount of content was unrealistic, since CSUN is a small IHE with limited 
resources, increasing class sizes and heavy faculty workloads. 

Thus, the internal conditions of CSUN, coupled with a familiarity with the passive lecture 
approach directly shaped the perception that an inquiry-based approach was not feasible, and 
thus not desirable. Furthermore, this respondent noted that in light of the constraints facing 
faculty and the aforementioned challenges with student learning, a single TNE hire would not 
change such large and multi-faceted problems. In other words, this respondent felt that the 
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TNE hiring effort was a relatively one dimensional response to a more eomplex problem that 
ineluded internal policies and external factors beyond CSUN’s control. 

• Post-reform (T2). There were no changes to this schema as a result of the SCALE project and 
other reform efforts. In fact, the failure of the TNE effort to hire a pedagogy specialist may 
have reinforced the respondent’s default schema favoring content over pedagogy, since there 
was a less than convincing display of their integration in a single lesson. 

Distinction between hard and soft sciences (CSUN-wide cultural schema). Responses 
are presented pre-, during, and post-reform. 

• Pre-reform (Tl). Even though this respondent had virtually no exposure to the learning 
sciences during graduate and post-doctoral training, and thus had little reference points to 
draw upon when thinking about the “soft sciences,” this “hard/soft” schema is so pervasive in 
academia that it underlay this respondent’s perceptions of STEM education reform. 

• During reform. Again, this schema was directly engaged by the TNE effort to hire a 
pedagogy specialist in the department. In describing the interview with a pedagogy- 
specialist, the respondent noted the following reaction; 

They didn’t strike as much of a chord with me as basic science. I don’t know if I harbor 
some bias that it’s not real research or something like that. I don’t really feel that way 
because I did get something out of the talks, but [there is] one thing. None of the people 
doing this research had anywhere close to the level of commitment or experience that all 
the rest of us had doing basic research. That’s what some of it boils down to — someone 
who’s been at it for a year or two, that’s nothing like what I had to do to get here — 6 
years of grad school, and 7th doing a post-doc — I had to invest so much more of myself 
(STEM faculty) 

This reaction suggests that even if the respondent did not harbor some bias towards 
educational research, the respondent had a predilection for basic science. This preference and 
familiarity, and its role as a filtering mechanism when faced with new information, is a 
classic instance of an individual’s schema asserting itself The respondent’s observation 
about skepticism was due to the applicant not having paid dues also hints at the notion that 
there is an apprenticeship period in academia, whereby individuals invest time, energy, and 
money and become accepted members of a disciplinary group. This notion is another 
manifestation of this schema, and points to the fact that sometimes cultural schema are only 
made visible during events such as the TNE hiring process. 

• Post-reform (T2). There were no changes to this schema as a result of the SCALE project and 
other reform efforts. In fact, the failure of the TNE effort to hire a pedagogy specialist may 
have reinforced the respondent’s schema that distinguishes between the hard and soft 
sciences. 

Tension between institutional support for reform and the disciplines (CSUN-wide 
cultural schema). Responses are presented pre-, during, and post-reform. 
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• Pre-reform (Tl). Given the aforementioned analyses on the dominance of discipline-based 
perspectives about the relative value of teaching and research, content and pedagogy, and 
views of the “soft” sciences, it is not surprising that this respondent also exhibited the 
cultural schema detailing the tension between STEM education reform and the disciplines. 
This tension was described less as an ideological conflict and more as one based on resource 
allocation and the fair treatment of faculty. 

Most of the faculty in our department are of the mind that this concept of a “Learning 
Centered University” is in direct conflict with the eroding resources provided by the state 
and that — I don’t know how else to say it — ^you can’t get blood out of a turnip. (STEM 
faculty) 

This perspective echoes other sentiments that efforts such as the ECU and TNE are in 
conflict with the realities of institutional life, and the cultural proclivities of academics whose 
first allegiance may be to their disciplines and their standards for research excellence. 

• During reform. This schema was directly engaged by the TNE effort to hire a pedagogy 
specialist in the department, which the respondent considered a failure. The department had 
brought in two candidates for on-site interviews, and neither had been considered sufficiently 
prepared for a tenure -track position. This estimation was made based on their lack of 
adequate content preparation (as noted above), which made this respondent skeptical that the 
candidates could handle the rigor of establishing an independent research program. As a 
result, the department decided to offer a non-tenure track position. In any case, this 
experience reinforced the notion that STEM education reform efforts like TNE were out of 
step with the criteria and demands of this particular STEM department. 

• Post-reform (T2). There were no changes to this schema as a result of the SCALE project and 
other reform efforts. In fact, the failure of the TNE effort to hire a pedagogy specialist may 
have reinforced the respondent’s schema. 

VI, Conclusions and Recommendations 

This section includes our conclusions and recommendations for program planners and 
policy makers regarding STEM education reforms. 

Assessing the Effects of SCALE at CSUN 

Without measurable objectives with which to evaluate the SCALE project’s activities and 
progress, it is difficult to make a definitive statement about the relative success or failure of the 
project. However, in light of the original goal to affect changes to undergraduate STEM 
instruction, interdisciplinary collaboration on preservice programs, interinstitutional 
collaborations on in-service programs, and the underlying institutional culture that informs these 
activities, the effects of the SCALE project must be considered modest. Of note, SCALE focused 
its efforts on a relatively small and discrete point within the CSUN context — engaging faculty to 
support K-12 teacher professional development. The SCALE project resulted in outcomes 
pertaining to the institutional context of CSUN that directly and indirectly relate to the four goals 
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of the project. This was possible largely because the SCALE initiative dove-tailed with other 
extant reform initiatives that collectively created a very favorable environment for STEM 
education reform. 

Below we summarize the key accomplishments of the SCALE project in terms of specific 
elements of the CSUN institutional context.'*’ By providing an account that situates the project’s 
outcomes within the complex institutional environment of CSUN, as opposed to a 
decontextualized list of project accomplishments, we hope to provide a more realistic and useful 
evaluation for program leaders and policymakers. Eigure 4 includes these outcomes in terms of 
the ICE, depicting relationships based on a causal network analysis, and noting a distinction 
between first and second-order outcomes. 

Effects of SCALE on the Institutional Context 

The following description of the effects of SCAEE on the institutional context is largely 
drawn from the preceding analysis of the science and math institutes (section III), and the 
cultural analysis in section IV. Outcomes listed below pertaining to pervasive beliefs and 
individual sense-making are described in detail in section IV. 

External environment. 

• SCALE developed four immersion units for LAUSD (first-order); 

• SCALE/QED held 13 science institutes and 4 math institutes between 2006 and 2007 (first- 
order); 

• SCALE provided professional development in math and science to 3 1 1 K-12 teachers (270 
science, 41 math) (first-order); 

• SCALE provided a more intensive type of interaction with K-12 than previously experienced 
(second-order). 

Internal structure. 

• Aspects of the SCAEE science institutes, including institute design and facilitation models, 
influenced CSP (second-order); 

• SCALE activities complemented the efforts of other reform initiatives, including the ECU 
and TNE initiatives (second-order); 

• SCALE became involved in the math department schism, (second-order). 



Professional development outeomes for K-12 personnel are not ineluded in this analysis as they did not direetly or 
substantively affeet UW-Madison personnel or institutional proeesses. 
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Resources. 

• SCALE expanded the STEM education cohort and provided professional development 
(second-order); 

• SCAEE provided funds (stipends and release time) in a resource-limited environment 
(second-order). 

Pervasive beliefs. 

• None 

Individual sense-making. 

• SCAEE engaged two STEM faculty participants as learners, shifting their roles in PD from 
that of content experts to a blend of content and pedagogy experts (first-order). 

Practices. 

SCALE provided at least three STEM faculty with pedagogical tools to use in their own 
courses (first-order). 

In addition, to understand the impact of SCALE on CSUN, bear in mind the relationship 
between the key contextual factors and institutional change processes identified in this study. 
Universities like CSUN are large, complex institutions comprised of many different, loosely 
coupled colleges and departments that largely operate according to their own logic, rules, and 
norms. As such, the contextual factors pertaining to STEM education identified in this study 
constitute a collection of disparate yet interconnected features that operate in dynamic interaction 
such that a change in one may yield unpredictable and even imperceptible movement in others. 

While SCALE did not create a revolutionary change at CSUN, it leveraged scarce 
resources (i.e., fiscal and human) to support activities that complemented existing reform efforts. 
In accidental coordination with the LCU, TNE, and CSP activities, it appears that SCAEE served 
to deepen the cumulative impacts that these projects alone may have made. These impacts were 
particularly felt in the further development of the STEM education cohort at CSUN. This raises 
the question about whether stronger outcomes may have been achieved if these disparate reform 
efforts had been strategically designed and managed according to an explicit theory of change. 
We speculate that to effect substantive change at CSUN, such a comprehensive strategy is, if not 
necessary, at least commensurate with the complexity and multi-dimensionality of the leverage 
points within the institution. 
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Figure 4. First- and second-order outcomes of SCALE activities at CSUN, 
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The Effects of a Project Implemented Through Distant Leadership 

Since there were no SCALE leaders present at CSUN, there was no explieit theory of 
institutional change. This situation strongly eontrasts with SCALE at both UW-Madison and 
CSUDH. SCALE engaged CSUN primarily as a site for its already existing math and science 
institutes, and largely ceded the effort at institutional transformation to chance and the extant 
reforms at CSUN. As previously noted, sinee the SCALE leaders at UW-Madison and CSUDH 
did not have deep local expert knowledge, the campaign approach to change used at their own 
IHEs could not be used effectively at CSUN. That is, there was essentially no connection 
between SCALE goals pertaining to institutional change and SCALE implementation strategy. In 
effect, SCALE planted its interventions at CSUN and relied on other loeal experts to conduet a 
ehange initiative that was explieitly linked to other reforms and whose operative theory of 
change at CSUN was less that of SCALE’S and more that of the ECU, TNE, and CSP projects. 
An enduring lesson from SCALE at CSUN is that multi-institutional reform efforts should focus 
on recruiting local leaders who are suffieiently informed about and active at eaeh partner 
institution. While SCALE was fortunate in engaging loeal leaders in K-12 districts (e.g., 
LAUSD) and some IHE partners, local leadership was lacking at CSUN. As previously noted, 
one effect of this situation was that there was no coordination between SCALE and other extant 
STEM edueation reform initiatives. This suggests that multi-institutional reforms should pay 
attention to eultivating local leaders instead of relying on a top-down, or in this case, externally 
managed, leadership strategy. 

Lessons Learned From the Cultural Analysis of CSUN 

Several core findings were derived from the analysis of the situated mental model of the 
CSUN faculty, and of the pervasive values and beliefs at CSUN pertaining to STEM education 
reform. These findings can be considered lessons learned from the interaetion between the 
models and STEM education reforms (i.e., SCALE, TNE, CSP, and the LNC initiatives), and the 
ultimate effects of these reforms on the models at the conclusion of SCALE. The following 
section includes the lessons learned from these analyses. 

There is a Cultural Complex of Scientific Legitimacy and Credibility 

Anthropologists sometimes describe elusters of similar praetiees (e.g., rituals, tool- 
making, languages) as “eultural complexes.” Eor example, archeologists have named a period in 
Wisconsin’s pre-history the “red ochre complex” for the common use of red ochre in burial sites 
throughout the central and southern parts of the state. Since these sites contained evidence of 
warfare and exotie artifaets, the eomplex refers not simply to the use of red ochre but also to the 
unique social, economic, and environmental contexts in whieh this practice developed 
(Birmingham & Eisenburg, 2000). 

Since the cultural schema pertaining to the beliefs and practices of STEM faculty were so 
eoherently related and embedded in speeific eonditions, we eonsider these schema and their 
related eontextual variables to be a cultural complex for scientific legitimacy and credibility. 

This complex is characterized by: 
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• Disciplinary prioritization of research, teaching, and service (STEM diseiplines); 

• Distinetion between hard and soft seiences that fuels long-standing tension between 
departments in the eollege of edueation and some STEM departments; and 

• Beliefs about the relative importanee of eontent and pedagogy in preserviee STEM eourses 

This cultural complex is partieularly resilient in part beeause it is developed through 
years of intense training and soeialization in eommunities of praetiee whose membership is 
earefully sereened (Lave & Wenger, 1991). As sueh, any STEM education reform effort should 
at the very least be eognizant of this eomplex and its unique instantiation at loeal sites. 

Mental Models are Hard to Change 

bindings reported in seetion III indieate that a few faculty who participated in the 
SCALE-led summer institutes experieneed some ehange in their views and teaehing praetiees. 
Essentially all who experieneed ehange were already disposed to make the types of ehanges 
sought by SCALE. Bearing these very limited outeomes in mind, while also eonsidering our 
analysis of one individual’s sehema, we speculate that it may be very diffieult to ehange most 
faeulty’s mental models for STEM edueation reform. This difficulty is consistent with the 
researeh on the difficulty of affecting behavioral ehange in edueation and the resilieney of 
culture (Stensaker & Norgard, 2001; Tobias, 1992). 

STEM edueation reform may tap into uniquely resilient and rigid eultural sehema that 
are part of the aforementioned eultural eomplex of seientifie legitimacy and credibility. If so, the 
mental models studied here may be unusually resilient and diffieult to ehange, in part beeause 
they have been learned over the eourse of several years of training and are integrated with loeal 
contextual faetors that serve to reinforce and reproduee them. 



Contextual Factors Shape Mental Models 

This analysis foeused on the role of eontextual faetors in how STEM edueation reform is 
interpreted, implemented, and ultimately effective in a unique institutional environment. One of 
the primary ways that the institutional context influenees reform initiatives is in shaping the 
mental models of faeulty. Eaetors that were partieularly influential for STEM education reform 
ineluded doetoral training, budgetary eonditions, institutional mission, leadership, soeial 
networks, RTF polieies, the eultural eomplex of seientifie legitimaey, and the influenee of the 
workload on individual sense-making. As sueh, mental models should not be viewed as eognitive 
struetures that operate in a vaeuum, as is the ease with lab-based researeh in eognitive seienee or 
psyehology. Instead, situated mental models operate in a real-world and eomplex setting whieh 
makes their study (and operation) more diffieult, and quite possibly, their manipulation more 
problematie. 
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Professional Communities are Critical in Culture Formation 

Some of the contextual factors most salient to mental models are the professional 
communities of which individual faculty are members. Some of these groups extend beyond the 
confines of a faculty member’s home institution and may include disciplinary societies, 
colleagues in a specialty area, and their alma mater. In any case, these social groups often 
develop their own particular language, behavioral norms, and belief systems that are particularly 
resilient (Lave & Wenger, 1991). 

A typical STEM faculty member’s professional communities are not likely to have strong 
ties to educational reform or the learning sciences. This is not true, however, for faculty who 
choose to actively engage in educational reform activities. In these cases, they identify groups of 
like-minded others, in their disciplinary society or at their home campus — as seen at CSUN. 
These groups may become important venues in which cultural schema that are supportive of 
educational reform develop, are reinforced, and reproduced to new members. In addition, these 
collegial groups provide moral support, practical advice, and a community of like-minded 
professionals. Faculty respondents involved in such groups explained that their education reform 
community was as engaging and legitimate for them as their own disciplinary communities. 

Cohorts of reform-minded faculty were enthusiastically encouraged by administrator 
respondents, who felt that these internal agents of change were far more effective than external 
consultants. Regarding interdepartmental relations within these communities, several education 
faculty respondents stated feeling that STEM faculty respected their training in pedagogy as 
much as their own content expertise. This went a long way in fostering collegial relations. 

A Comprehensive Approach to Culture Change and Culture-Brokers is Important 

This analysis suggests that shaping the culture of an organization may require 
comprehensive efforts to change the structural, social, and symbolic milieu in which individuals 
operate, in addition to the actual cognitive processes that constitute individuals’ habits of mind. 
This may require leaders to employ a flexible and multi-faceted toolkit of frames through which 
to analyze their organizations (Bolman & Deal, 2003). In addition, such an effort would benefit 
from a deliberate and strategic approach by reform or campus leaders, as opposed to an unguided 
intervention that unfolds according to chance. It is especially important that a local figure play a 
leadership role, as mental models may operate according to a logic that is inaccessible to external 
leaders and change agents. In cases where inter-institutional or interdisciplinary collaborations 
are taking place, it may also be important for a local individual to play the role of a culture- 
broker. Our case studies of UW-Madison and CSUDH indicate the importance of such an 
individual, who understands the cultural schema operative within different groups, and can 
carefully negotiate the divisions or tensions that may exist between the groups and their 
pervasive values and beliefs. 

Key Leverage Points for Institutional Change 

We judge that the following seven contextual factors identified in this analysis functioned 
especially effectively as key leverage points for affecting the types of change sought by the MSP 
program: 
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1 . Points of contact with the external environment; 

2. Leadership at all levels; 

3. Decision-making bodies and interdepartmental forums; 

4. Networks of STEM edueators; 

5. Other reforms; 

6. Cultural complex regarding disciplinary legitimacy and credibility; and 

7. Individual faculty workloads 

We identified these points by foeusing on: (a) factors that were directly linked to the first 
and seeond-order outeomes of SCALE activities, and (b) factors that had particularly strong 
downstream effects on the institutional context of CSUN. The multi-dimensional framework 
used in this case study shows that these factors, which we will henceforth refer to as leverage 
points, inelude, but are not limited to, faetors eommonly eited in the change literature, such as 
policy initiatives, school leadership, or professional eommunity. Eurthermore, we note that the 
leverage points identified in this study should not be viewed as isolated magic bullets that can 
produce fast and enduring reform across the entire university. Rather, they must be understood as 
operating in dynamie interaction, such that a change in one may yield unpredietable and even 
impereeptible movement in others. 

Based on our analysis of the SCALE project, we speculate that by engaging the key 
leverage points described below in a substantive and sustained manner, it is possible that a 
tipping point may be reaehed within a partieular organizational unit, sueh that a previously rare 
practice or behavior becomes widespread (Gladwell, 2000). This analysis is intended only to 
identify the key leverage points for change in this system, and no claims are made regarding the 
ultimate outcomes on instructional practices or student learning. We suggest that the leverage 
points below deserve partieular attention by eampus leaders and STEM edueation reformers, 
sinee they have the potential to effeet a easeade of impacts in a variety of points in the cultural 
context of an IHE. Figure 5 presents these leverage points in the context of the external and 
internal environments of CSUN, illustrating that these leverage points operate aeross many levels 
of this complex organizations system. 

Points of Contact With the External Environment 

An IHE has numerous points of contact with the external environment, including 
accreditation agencies, local K-12 districts who supply students, other IHEs who train future 
faculty, and government policymakers, to name but a few. In many cases, exogenous pressures, 
as opposed to internally derived deeisions or motivations, are a significant source of the stimuli 
that require institutional change. While many points of contact with the external environment are 
beyond the control of CSUN, it is possible to anticipate how these stimuli will influence the 
institution and act accordingly. For example, respondents noted ways in which the university had 
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The External Environment 




1 = Points of contact with the external environment 

2 = Leadership at various levels 

3 = Decision-making bodies and inter-departmental forums 

4 = Networks of STEM educators 

5 = Other STEM education reforms 

6 = The cultural complex for disciplinary legitimacy and credibility 

7 = Individual faculty workloads 

Figure 5. Key leverage points for institutional change at CSUN, 

been proactive and responsive to changing environmental conditions. An example is the response 
to a limited pipeline of preservice science majors between local K-12 districts, community 
colleges, and CSUN. Faculty active in the TNE project conducted a study to assess how local 
high school students felt about K-12 teaching careers, so that they could adapt their recruiting 
programs to address their preconceptions, fears, and career hopes. In this way, CSUN was able to 
respond to an external condition, analyze it, and possibly tailor its internal activities to address 
the condition’s effect on the institution. 



Leadership at all Levels 

Educational leaders serve a variety of functions, including establishing policies and 
procedures, setting goals and priorities for the institution, and establishing a particular tone 



63 




A Final Case Study of SCALE Activities at CSUN 



through communications with staff. As such, leaders play a vital role in creating an institutional 
environment in which change can take plaee (Kezar & Eckel, 2002). In the case of CSUN, 
leadership at the upper administration level was critical in establishing the ECU initiative and 
eonveying an unmistakable signal to faculty that pedagogical reform was a high priority. In some 
eases, leadership at the eollege and departmental levels operated according to a different set of 
priorities, which may have hindered the upper administration’s efforts. This scenario is largely 
due to the governance structure at CSUN, and is a predicament for most change initiatives in 
higher edueation. Ideally, when faced with an institutional change initiative like the ECU, some 
sort of compromise eould be made among leadership at all levels of an IHE so that their 
respective needs and priorities could be addressed. Such an approach could also serve to 
minimize the likelihood of a disjuncture among leaders, some of whom may act (consciously or 
not) to undermine the ehange effort. 

Decision-making Bodies and Interdepartmental Forums 

Certain deeision-making bodies, such as the liberal studies committee at CSUN, may be 
eritieal to reform efforts such as SCALE, as they are pre-existing venues for inter-departmental 
collaboration that meet for specific reasons of common interest, and generally have decision- 
making authority. These organizational units strongly influence the structural factors that shape 
undergraduate edueation, and eomprise the policy environment that individual instructors must 
respond to when planning and teaching math and science eourses for preserviee teachers. It 
appears that any STEM education reform effort at CSUN aimed at affecting lasting policy 
changes would need to engage this leverage point. In contrast, committees or working groups 
established during a grant period may not appeal to a broad spectrum of faculty, and may be less 
likely to become institutionalized. Furthermore, at a campus as large and deeentralized as CSUN, 
the status quo is that faculty in different departments do not interact in a sustained, goal-directed 
manner unless individual faculty initiate interdepartmental forums. With regard to achievement 
of goals sueh as those SCALE seeks, sueh forums may provide eritieal leverage points. 

Networks of STEM Educators 

Faeulty belong to a variety of professional and personal networks, which in some eases 
can become professional communities if they involve meeting regularly and addressing common 
problems and tasks. These networks and communities are a key leverage point because through 
them ideas can spread rapidly, resources can be accessed, and collegial support for new 
endeavors can be obtained. Partieularly important members of these STEM educator networks 
are faculty and staff who have positions focused on STEM education, which is not uncommon at 
CSUN. STEM faculty who are already engaged in these activities as part of their normal 
workload are another key leverage point for reform, as they are well-plaeed to enact change and 
do not have to negotiate their workload in order to focus on STEM education issues like other 
colleagues must. Furthermore, these faculty and staff often have extensive networks and access 
to STEM education resources that can be utilized to achieve the goals of a reform initiative. It is 
important to maintain contact with these faculty and staff, and to understand the dynamies of 
their particular departments regarding STEM education, in order to avoid inadvertently 
jeopardizing their reputations or careers. 



64 



A Final Case Study of SCALE Activities at CSUN 



Other Reforms 

At any given time several pedagogical reform initiatives may be underway at CSUN or at 
any other IHE. Some of these may be high-profile initiatives such as TNE, or smaller efforts led 
by individual faculty. Historic and active reforms offer an important resource through 
establishing social networks, lines of communication, policies and procedures, and trained 
faculty or staff. For example, a respondent noted that Dr. Oppenheimer’s decades of work with 
local K-12 districts had led to strong relationships with individual K-12 administrators and 
teachers that newer faculty were then tapping into. However, a panoply of efforts may also result 
in an uncoordinated set of activities, each of which only has a local effect. Thus, it may be 
important to harness the energies and resources of disparate activities into a coherent whole, 
including standardizing evaluation procedures or explicitly nurturing networks of STEM 
educators across projects. 

Cultural Complex Regarding Disciplinary Legitimacy and Credibility 

Given its resiliency and dominance in academia, the cultural complex for disciplinary 
legitimacy and credibility must be first recognized, and if possible, exploited by leaders of 
reform initiatives. Achieving legitimacy and credibility in one’s field provides two possible 
opportunities regarding STEM education activities. First, it affords faculty with safety from 
colleagues’ criticism and a strong sense of confidence that is important when departing from the 
traditional workload of a STEM faculty member. As one respondent stated, if someone’s 
research credentials are solid then they can dabble in less prestigious issues such as education. 
Second, once faculty achieve tenure and are under less pressure to publish and conduct research, 
they many have more time to explore other ideas and activities. 

Individual Faculty Workloads 

Perhaps the most important leverage point for institutional transformation is the daily workload 
of faculty and staff, which is generally overwhelming and is comprised of demands from a 
variety of constituencies (e.g., students, colleagues, administrators, funders). This reality must be 
acknowledged and addressed before faculty can be fruitfully engaged and possibly convinced of 
the need for an institutional reform. Otherwise, a reform effort may antagonize a workforce that 
is already under significant pressure. This critical factor may be addressed by buying out 
faculty’s time, ensuring that work on pedagogical reform will be valued by RTP committees, or 
aligning the goals of reform with pre-existing work responsibilities, (e.g., see leverage point 3, 
above). 

Recommendations for CSUN Leaders and the NSF 

This case study, combined with our case studies of UW-Madison and CSUDH, reveals 
mechanisms of change initiated by a STEM education reform effort at an IHE, and in the 
process, illuminates an enacted theory of change that appears to have achieved at least some of 
its intended outcomes. We consider this theory of change promising, with the caveat that 
effective implementation of the theory requires a sophisticated understanding of the multi- 
faceted nature of the barriers and supports within an IHE. The following recommendations are 
based on this theory of change and include a set of core concepts that may (a) help CSUN leaders 
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to continually improve their efforts, and (b) help the NSF, Department of Edueation, and other 
agencies design poliees that more effeetively foster aehievement of MSP goals for IHEs. 



Eor reform efforts that target institutional eulture and individual eoneeptual ehange, 
partieular attention should be paid to how the intervention spaee is designed sueh that the unique 
characteristies of a local site (and its organizational membership) are engaged in a fruitful 
manner. Based on the findings of our three ease studies, the following points should be 
eonsidered when developing sueh a reform: 

Conduct Assessments of the Institutional Context Prior to Program Planning and 
Implementation 

We propose that ehange efforts should begin with an institutional needs assessment in 
order to identify the loeal eontextual faetors that may provide barriers and opportunities to 
reform (Tobias, 1992). Treisman (2007) suggests that this is important so that ehange leaders 
obtain a “elear sense of the idiosyneratie features of the environment” (Treisman, 2007). Sueh an 
institutional diagnosis should aeeount for the different aspeets of the institutional eontext as 
detailed in this ease study, ineluding the external environment, internal strueture, resourees 
(fiscal and social), pervasive values, individual sense-making, and praetiees. One of our next 
steps is to develop a diagnostie tool that is eost-effeetive and easily administered so that program 
planners ean tailor their programs to their institutional eontexts (see below), and also establish 
baselines for evaluation purposes. 

Ensure that Recruitment Efforts Pay Attention to Workload and Cultural Factors 

Reform leaders should address and aeknowledge the eurrent aetivities, eonstraints, and 
pressures faeing STEM faeulty when reeruiting and engaging them in reform efforts. For 
example, when presenting a reeruitment piteh, begin by aeknowledging the suceesses and present 
eonditions of a particular department and then present the ehallenges that the reform effort hopes 
to address. This strategy may help reform leaders avoid appearing adversarial or out of toueh 
regarding the day-to-day eoncerns of many faculty. This “meet them where they are” stanee 
would also be useful during projeet implementation. In addition, faeulty and staff need a 
rationale to beeome engaged in a reform effort. Altruism or a sense of responsibility may not be 
enough, partieularly if the workload pressures are substantial. Thus, a reform effort should 
explore ways to mitigate the workload by providing release time for faeulty, or targeting senior 
faeulty whose workload pressures may be less demanding than their junior eolleagues. 

Reform efforts should pay partieular attention to the eultural eomplex of seientifie 
legitimaey and credibility, sueh that the reform does not antagonize an individual’s diseiplinary 
identity and related mental models. If this faetor is not addressed, reform leaders run the risk of 
immediately putting off potential partieipants, whieh is not uneommon. Thus, reform leaders 
should eonsider tailoring their reeruitment efforts sueh that this eultural eomplex is explieitly 
addressed by having high-prestige faculty serve as reeruiters, and ensuring that the rhetorie used 
in program reeruitment and materials is not overly eombative or filled with teehnieal terms from 
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education research. The use of educational jargon can serve as an immediate trigger to the 
cultural schema distinguishing between hard and soft sciences. 

Design Neutral Spaces in Which Different Groups may Interact 

Reform initiatives should also focus on designing neutral spaces in which different 
constituencies can safely and productively interact. This may entail paying close attention to the 
symbolic nature of meeting spaces or particular facilitators, and the power inequalities that may 
exist between different groups (e.g., STEM faculty, education faculty, K-12 staff). For example, 
interdisciplinary forums where STEM and education faculty meet should allow for the 
productive exchange of ideas, and account for the potential conflict inherent in the cultural 
schema that distinguishes between hard and soft sciences (see above). Neutral space can also 
ensure that all participating faculty and staff are integrated into the reform effort, such that 
participants feel that their expertise and workload are both being valued and addressed. 

Recruit a Skilled Culture-Broker When Working with Interdisciplinary Groups 

When working with individuals from different institutional or disciplinary backgrounds, 
it is helpful to have a skilled facilitator, or culture-broker, who can negotiate the potential 
divisions and cultivate the points of cognitive and practical overlap that may exist among groups. 
In this case, critical leverage point in altering faculty members’ mental models of STEM 
education reform may entail surfacing of their assumptions about STEM education reform, and 
encouraging them to think like novices — processes likely to be accomplished through skillfully 
facilitated experiences. 

Marshal Existing Resources and Reform Projects to Collectively Target Key Leverage Points 

The large number of STEM education reform leaders and existing projects at CSUN 
represent a significant pool of resources and institutional knowledge that could be more 
effectively marshaled. At the present time, these projects are loosely coordinated, if at all, due to 
their different missions and the sheer size of IHEs. However, this should not keep reform leaders 
from meeting on a regular basis to share notes, combine resources when advantageous, and when 
appropriate, collectively leverage existing political, social, and fiscal capital to exert pressure on 
strategic points in the system of STEM education. 

Focus on Developing Cohorts of STEM Educators in Specific Departments 

We also recommend that clusters of faculty with STEM education expertise be recruited 
in specific departments in order to achieve critical mass and minimize departmental resistance to 
pedagogical change. Whereas an individual change agent working in a hostile or disinterested 
institutional environment is unlikely to convince colleagues to change policies or practices, a 
small group of colleagues amenable to reform has a chance to effect change. In addition, the 
STEM education experts would benefit from having a supportive community of colleagues with 
which to share ideas and resources. 
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Furthermore, instead of foeusing exclusively on engaging STEM faculty, which is the 
wont of current reform initiatives such as the MSP, we recommend an even more deliberate 
focus on particular groups of faculty and staff. The following four groups represent particularly 
rich opportunities for programs such as the MSP: (a) STEM and education faculty who are 
already engaged in educational activities; (b) academic staff who already are engaged in 
educational activities; (c) STEM and education faculty who occupy key administrative positions 
relevant to STEM education; and (d) STEM and education graduate students. 

Carefully Design Top-Down Structural Reforms With Attention to the Moving Parts of the 
Institution 

Campus leaders should consider potential policy levers, such as requiring faculty to 
assess student learning outcomes, as leverage points to support campus-based efforts. However, 
these efforts should be carefully designed to be responsive to the multifaceted contextual factors 
operative at a particular institution, including existing policies, pervasive values, and resources. 
As this case study suggests, particular attention should be paid to the mental models, and their 
constituent cultural schema, of individuals likely to be impacted by a reform. Research on the 
implementation of reforms fostered from the top shows that these efforts fail largely due to 
resistance of local agents, whose interests, needs, and cultural norms run counter to the perceived 
intent of the reform (Spillane, Reiser, & Reimer, 2002). An institutional needs assessment (see 
above) focused on identifying the cultural conditions of administrative units most likely to be 
affected by policy change can provide this type of understanding. 

Next Steps 

This line of work for the SCAEE project includes case studies of two other IHEs, 
CSUDH, and UW-Madison plus a final cross-case analysis for all three sites, written for a 
practitioner audience in mind. This cross-case analysis of SCAEE activities at its IHE partner 
sites will focus on further developing a diagnostic approach to evaluating STEM education 
interventions in complex organizations. In particular, this analysis will explore the prospects of 
linking changes in the institutional context to changes in instructional practices and ultimately, 
student learning outcomes. 
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Appendix 

Methodology 

This research used a repeated cross-sectional qualitative case study. The qualitative case 
study design was selected due to its utility in conducting empirical inquiry into a “contemporary 
phenomenon within its real life context, especially when the boundaries between phenomenon 
and context are not clearly evident” (Yin, 2003, p. 23). This design seemed particularly 
appropriate for highlighting contextual factors that affect an MSP project’s implementation 
processes and outcomes. Moreover, its focus on how and why is expected to yield valuable 
information for program funders and planners (Owen & Lambert, 2001). We selected this 
design — in contrast to a longitudinal study that would follow a single cohort who participated in 
a single intervention — because of the emergent nature of the SCALE project and the reality of 
participant attrition in projects of this nature. We decided to capture the effects of a large number 
of project activities and their varied participant populations, understanding that we were 
sacrificing the ability to attribute program effects to a specific intervention in a robust fashion. 

Sampling Procedures 

The primary unit of analysis for this research is the individual, and the sampling universe 
for this research included all CSUN personnel. We then used two sampling frames to identify 
participants in the study: (a) participants in any SCALE activity, and (b) personnel in 
administrative units that were the focus of SCAEE, including STEM and education departments, 
administrators, and academic staff. Nonrandom sampling procedures were used to identify 
SCAEE participants, and snowball sampling was used to identify nonparticipants who differed 
with regard to their experience with STEM education reform. Nonparticipants with little 
experience in STEM education reform were included to test and/or confirm findings from the 
SCALE participants, as the latter may constitute a biased sample regarding their perceptions of 
the institutional context. 

Data collection took place at two points in time: Time 1 (Tl; June 2006) and Time 2 (T2; 
June 2007). At Tl a total of 18 interviews were conducted, and at T2 a total of 16 interviews 
were conducted (see Table Al). Due to respondent unavailability and faculty turnover (at both 
CSUN and with SCAEE) only 5 SCAEE participants were interviewed at both Tl and T2. In 
addition, we draw on interviews conducted with SCALE leaders and UW staff, conducted for the 
CSUDH and UW-Madison case studies to collect information on program planning, 
implementation, and evaluation. 



Sources of Evidence: The Data 

The data collected for this study are in-depth interviews, documents, and field 
observations of meetings and informal settings. The interviews were semi-structured using a 



” At Tl another researcher conducted 7 interviews in early 2006. A similar interview protocol was used by both 
researchers. 
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Table A1 

Numbe r and Role of Respondents at Time 1 and Time 2 





T1 (June 2006) 


T2 (June 2007) 




Total 

respondents 


SCALE 

participants 


Total 

respondents 


SCALE 

participants 


STEM faculty 


7 


4 


10 


8 


Education faculty 


6 


1 


4 


2 


Administrators 


5 


0 


2 


0 


Sub-Totals 


18 


5 


16 




Total respondents 


18 


16 



standard interview protocol for different types of respondents (i.e., STEM faculty, education 
faculty, administrators, etc.)- 

Ethnographic Interviews 

The interview protocol focused on eliciting how respondent approaches to the SCALE 
goal areas (i.e., STEM instruction, interdisciplinary and inter-institutional collaboration) are 
related to their institutional environment, if at all. 

Eollowing James Spradley (1979), we use the “ethnographic interview” method that, 
instead of rigidly adhering to a structured interview protocol, focuses on eliciting native 
terminology and topics of interest that may guide the line of questioning. While we were guided 
by a structured protocol in the manner advocated by Spradley, we used subtle probes to ensure 
that respondents identified the relationships between their sense-making, tacit assumptions 
(cultural schema), and the institutional context. For example, one question was: “What factors 
influence your participation in preservice programs?” Responses may include references to state 
policies, departmental teaching assignments, or personnel preferences. We then probed and 
asked precisely how state policies relate to their participation in preservice programs, and so on. 

Using these interviewing methods we were able to elicit two types of information used to 
explore the relationships among individual-level cognition, group-level culture, and the 
institutional context. First, we used the relationships between factors identified by respondents as 
input for the causal networks, which graphically depict how factors at each level dynamically 
interact in a manner that either supported or inhibited achievement of SCALE goals (see below). 
Second, to obtain data for our analyses of cultural schema, we asked specific questions about 
tacit assumptions pertaining to the SCALE goal areas, although often we found it necessary to 
infer these assumptions. In either case, we probed by asking if the belief or value was shared by 
any group of which they were a member. In this way, we were able to ascertain with which 
group the assumption, belief, or value could be associated. 

Documents and Field Observations 

Documents related to the university were also collected and analyzed, including reports 
from the university’s Office of Institutional Research, strategic plans, external evaluations of 
related programs, and RTF policies. Documents were identified by both respondents and 
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researchers and were analyzed at both T1 and T2. Field observations of the research site were 
also conducted, including three SCALE-related meetings, informal observations of respondents’ 
work environment, and of public presentations of the SCALE project. Einally, the preliminary 
case study of SCALE activities at ETW-Madison (Hora & Millar, 2006) provided the analytic 
foundation for the present study. 



Data Analysis 

The analytic procedures for this research draw on established procedures of qualitative 
analysis. These include inductively coding interview transcripts using the grounded theory 
method of Strauss and Corbin (1990), causal network analysis that graphically organizes the data 
by time and sequence (Miles & Huberman, 1994), and an exploratory analysis of cultural schema 
(Strauss & Quinn, 1998; Quinn, 2005). 

Coding Interview Transcripts Using a Grounded Theory Approach 

Using the findings from preliminary analyses of data from all three IFlEs, we developed a 
coding paradigm called the institutional context framework (ICE). A coding paradigm is a 
structured coding scheme used to analyze data and identify discrete themes and patterns (Strauss 
& Corbin, 1990; Ryan & Bernard, 2003). A coding paradigm is especially necessary in this 
instance in order to categorize and reduce the multidimensional data we had collected. The 
classification framework is organized into six broad categories that include more specific topics 
that are sub-codes used in our analysis (see below). It is important to note that the categories 
used to code the data are derived from analyses of complex institutional environments that are 
exclusively focused on STEM education, teacher preparation, and IHE/K-12 partnerships (Hora 
& Millar, 2006). As a result, it is possible that the framework adequately models only categories 
and topics related to SCALE goals. 

The six broad categories of the ICE are listed below. 

1 . External influences: Institution type, national and state education policy, academic training 
of faculty, economic forces affecting education, and local K-12 characteristics. 

2. Internal structure: Organizational structure (governance, teacher education programs, STEM 
degree programs), student body composition, instructional workforce composition, personnel 
policies, leadership, and reform initiatives. 

3. Resources: Material resources (e.g., time, funding), and social resources (e.g., networks). 

4. Collective beliefs and values: The beliefs, values, and tacit assumptions operative among 
groups at UW-Madison, including the institution as a whole, colleges, departments, and 
smaller communities or “sub-cultures.” 

4. Individual disposition and sense-making: The primary elements of an individual’s sense- 
making process, including workload considerations, personality, background and training, 
views on instruction, and status. 
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5. Practices: An individual’s classroom instruction (e.g., planning and delivery) and task-based 
collaborative activities with both IHE and K-12 partners. 

We then developed a three-step coding system that consisted of the following: (a) the ICF 
eategories; (b) topics considered barriers or enablers to SCALE as identified by either the 
respondent or the analyst; and (c) changes attributed to SCALE aetivities. This coding structure 
was designed to facilitate later data queries that would identify relationships among factors, 
particularly if a category in the ICF was considered a barrier or an enabler to SCALE goals. We 
then eoded both T1 and T2 interviews using NVivo qualitative analysis software, and conducted 
queries of the eoded interviews as a first step in identifying the themes for the case study. 

Queries were conducted based on high frequencies of eode sources and references (as 
seen in Table A2), and analyst judgments regarding the importance of the topic based on field 
notes and analytic inference. In this report sourees refers to the number of interview transcripts 
that include the eode, and referenees inelude the aetual number of codes applied to the 
transcripts. 



Table A2 

Summary Report of NVivo Coding 



Most Frequently Used 
ICF Codes 


Number of Sources 


Number of References 


External Influences 


Local K-12 Characteristics 


23 


60 


Institution Type 


20 


56 


Field of Higher Ed 


20 


48 


Alignment 


6 


15 


Edueation Policy 


19 


33 


Economics 


7 


13 


Doctoral Training 


13 


18 


Individual Perspectives & Sense-making 


Learning & Pedagogy 


30 


231 


Workload Considerations 


26 


70 


Legitimacy & Credibility 


25 


80 


Colleagues 


32 


285 


Teacher Ed 


25 


124 


K-12 


27 


113 


Epistemology of Disc 


21 


34 


Home Institution 


31 


280 


Background & Training 


29 


104 


Student Body 


26 


94 


Primary Motivations 


30 


160 


Career Status 


23 


66 


Internal Structure 


Teacher Ed Programs 


22 


88 


Change Proeesses 


25 


96 


Hiring & Personnel Policies 


30 


121 
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Most Frequently Used 
ICF Codes 


Number of Sources 


Number of References 


Workforce Topics 


15 


33 


Reform Initiatives 


30 


250 


STEM Degree Programs 


14 


27 


Leadership 


23 


57 


Influence of Location 


13 


27 


Student Body Topics 


22 


67 


Politics & Power Relations 


19 


36 


Practices 


Rationale Behind 
Instruction 


28 


93 


Lesson Delivery 


16 


31 


Inter-College Collaboration 


29 


116 


Inter-Institutional 

Collaboration 


13 


23 


Research & Publishing 


27 


69 


Engagement w/ Pre-service 


24 


80 


Committee Work 


18 


35 


Service 


20 


50 


Resources 


STEM Ed Community 


10 


33 


Eunding 


25 


94 


University-wide Colleagues 


11 


27 


Time 


27 


71 


Department Colleagues 


12 


19 


Shared Meanings 


Department Norms 


29 


129 


Distinctions Between 
Groups 


31 


172 


University Norms 


26 


103 


Individual in Relation to 
Sub-Group Norms 


26 


60 



Next, we eonducted matrix queries to seareh the coded interviews for specific 
combinations of codes; in this instance we were most interested in text that had been double or 
triple coded with ICF categories, the second pass (barriers/enablers), and the third pass (changes 
attributed to SCALE). Coarse category results are presented in Table A3. 
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Table A3 

Sample NVivo Matrix Query for First Pass (Internal Structure) and Second and Third Pass 
Codes 



Changes attributed 

Enablers to SCALE Barriers to SCALE to SCALE 



1 : Teacher Ed Programs 


19 


'26 


16 


2 : STEM Degree Programs 


14 


22 


6 


3 : Governance 


7 


17 


3 


4 : Instructional Workforce 


14 


20 


8 


5 : Personnel Policies 


9 


29 


7 


6 : Leadership 


9 


9 


5 


7 : Student Body Topics 


8 


14 


3 



It is important to note that frequeneies alone were not sufficient for identifying key 
factors and/or findings, as the codes were sufficiently broad to allow for a range of perspectives. 
It was necessary to conduct a more in-depth analysis of the coded text in order to ascertain the 
specific nature of the factor under consideration, which we did by reviewing all of the coded text 
and actively seeking disconfirming evidence from the interview transcripts and other data 
sources (Miles & Huberman, 1994). Once the most frequently cited factors had been identified, 
we then assigned a valence in order to denote a positive (+) or negative (-) influence on SCALE 
activities, based on a respondent’s indication of the nature of the factor, or our own analysis of 
the factor and its relationship to SCALE goals. Each weighted factor was then linked to those 
SCALE activities to which respondents associated it. Again, we constantly referred back to the 
data when making these assignments and actively sought disconfirming evidence in the 
interviews and documents. We then used the following criteria to evaluate if a factor would be 
included in the final analysis: (a) document-based evidence of institutional phenomena, SCALE 
activities, or SCALE outcomes; and (b) at least three respondent accounts of institutional 
phenomena, SCALE activities, or SCALE outcomes. The resulting factors constitute the primary 
data for this case study. 

Causal Network Analysis: Situating the Factors Within a Time-Ordered Display 

Using the results from the coding, we organized the factors in a time-ordered display in 
order to identify relationships between factors at T1 and T2, and how SCALE mediated the 
implementation process between the two points. A causal network is “an abstracted, inferential 
picture organizing field study in a coherent way” (Miles & Huberman, 1994, p. 153). Eactors 
identified in the data are located on the y-axis according to where they are categories within the 
ICE, and on the x-axis (which denotes time) according to whether they occurred at TI, T2, or as 
a SCALE activity. 



Exploratory Cultural Schema Analysis 

Working within the interpretive tradition of Strauss and Quinn (1998) that uses natural 
discourse to identify cultural models, we analyzed the cultural schema for math instruction held 
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by two key subgroups active in the SCALE project. We strongly emphasize that this is an 
exploratory effort to identify cultural schema, undertaken in order to understand the role of 
culture in STEM education. First, we analyzed text coded for the ICE category “collective values 
and beliefs” for all respondents. When at least three respondents expressed a similar value or 
belief pertaining to STEM education reform, we classified it as a cultural schema. Using this 
process, we identified various cultural schema, some of which were shared among certain groups 
(e.g., biologists). 

Second, we analyzed if and how these schema were engaged or illuminated during the 
activity for one faculty member. This evidence is based on the respondent’s accounts of 
experiences and perceptions during the activity, and how personal beliefs, or counterpart’s 
beliefs, values, and tacit assumptions were made visible. 

Third, we analyzed changes in the respondent’s cultural schema and/or personal mental 
models that could be attributed to the activity, either by the respondent or by analytic inference. 
Our focus on personal mental models when analyzing project outcomes, as opposed to the sub- 
group’s cultural model, was necessitated given the small sample size. Our assumption is that any 
changes due to the SCALE intervention pertain not to the cultural schema of this group but only 
to the individual respondent. 
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